Add docs/tartube-spec.md: structured analysis of Tartube's codebase
To make the "surpass Tartube" roadmap concrete, document Tartube's v2.5.231 codebase as a specification we can refer to: - Module map: 146k lines of Python across 19 modules, with per-module responsibilities (mainapp / mainwin / config / downloads / media / options / wizwin / process / refresh / tidy / info / updates / ffmpeg_tartube / formats / ttutils / dialogue / files / xdg). - Data model: container hierarchy (Folder → Channel | Playlist → Video) with N-level Folder nesting, system folders that act as smart filters (Bookmarks, Favourites, Waiting, New, Live, Missing, Recent, Temporary), and the full per-video flag inventory. - OptionsManager: the headline feature — 164 distinct yt-dlp flags attachable to any container or video with cascading resolution. The spec lists the option groups (Behaviour, Network, Playlist, Filters, Download, Filesystem, Auth, Anti-bot, Format, Subtitles, Post-processing, Cookies, Output, Tartube-specific) and points to a concrete SQLite + Rust struct design for our Phase 1.1. - Seven threaded operation managers (Download, Update, Refresh, Tidy, Info, Process, Livestream/Stream) and three downloader strategies (VideoDownloader, ClipDownloader, StreamDownloader). - Custom downloads (named recipes) and profiles (saved container groups) — feeding our Phase 1 + 3. - UI surface inventory: five primary tabs (Videos, Progress, Classic Mode, Drag and Drop, Errors), ~80 specialised dialogs, three wizards (Setup, Import YT subs, Tutorial), and the ~30-tab SystemPrefWin. - SponsorBlock + chapter persistence (slice_list / stamp_list) — a Phase 1 add-on after per-channel options ship. - Notable corners: master/slave dbid dedup, RSS-driven discovery, external_dir per-channel storage, livestream message parsing, full gettext i18n. - Things Tartube does better (15 items, mapped to roadmap entries), things we already do better (10 items, to keep momentum honest). - A 12-checkbox "Tartube parity" milestone definition — when all are ticked, the comparison table at the top of ROADMAP.md goes from 8-ahead-8-behind to 8-ahead-0-behind, and "surpass" begins. ROADMAP.md gains a pointer to the spec just under the north-star line. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Luna
parent
555e0e1574
commit
c713d05db8
2 changed files with 622 additions and 0 deletions
|
|
@ -4,6 +4,11 @@
|
||||||
|
|
||||||
**Surpass [Tartube](https://github.com/axcore/tartube) in every dimension.**
|
**Surpass [Tartube](https://github.com/axcore/tartube) in every dimension.**
|
||||||
|
|
||||||
|
A structured analysis of Tartube's codebase, data model, operations, and
|
||||||
|
configuration surface lives at [`docs/tartube-spec.md`](docs/tartube-spec.md).
|
||||||
|
It enumerates the exact features we need to match — every Phase 1 item
|
||||||
|
below traces back to a specific Tartube subsystem.
|
||||||
|
|
||||||
Tartube is the mature open-source yt-dlp GUI and the obvious benchmark for a
|
Tartube is the mature open-source yt-dlp GUI and the obvious benchmark for a
|
||||||
project in this space. yt-offline has architectural advantages Tartube can't
|
project in this space. yt-offline has architectural advantages Tartube can't
|
||||||
catch up on quickly (Rust + axum + a real web UI + bundled toolchain + a
|
catch up on quickly (Rust + axum + a real web UI + bundled toolchain + a
|
||||||
|
|
|
||||||
617
docs/tartube-spec.md
Normal file
617
docs/tartube-spec.md
Normal file
|
|
@ -0,0 +1,617 @@
|
||||||
|
# Tartube Specification
|
||||||
|
|
||||||
|
A structured analysis of [Tartube v2.5.231](https://github.com/axcore/tartube)
|
||||||
|
(2026-05-24), our project's primary feature benchmark. Written to give
|
||||||
|
yt-offline a concrete target to work toward — every section ends with a
|
||||||
|
"What we can do" note about whether to adopt, adapt, or skip.
|
||||||
|
|
||||||
|
Tartube codebase: ~146,000 lines of Python 3 across 19 modules under
|
||||||
|
`tartube/`, plus locale, icons, sounds, screenshots. GTK 3 frontend.
|
||||||
|
GPL-3-or-later. Single-user desktop application.
|
||||||
|
|
||||||
|
## 1. Module map
|
||||||
|
|
||||||
|
| File | Lines | Responsibility |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `mainapp.py` | 30,565 | The `TartubeApp` Gtk.Application — owns 1,601 instance variables / 791 boolean toggles, every config field, every operation entry point. |
|
||||||
|
| `mainwin.py` | 42,074 | `MainWin` Gtk window + ~80 specialised `Gtk.Dialog` subclasses + the Video Catalogue widgets. |
|
||||||
|
| `config.py` | 36,659 | Preferences and per-object editor windows (per-channel options, FFmpeg options, scheduled actions, custom-download recipes). |
|
||||||
|
| `downloads.py` | 12,188 | Download orchestration: `DownloadManager` (worker pool), `DownloadList` (queue), `DownloadWorker`, three downloader strategies (`VideoDownloader`, `ClipDownloader`, `StreamDownloader`), JSON fetchers. |
|
||||||
|
| `ttutils.py` | 5,015 | Cross-cutting helpers: path manipulation, livestream-message parsing, locale, sorting. |
|
||||||
|
| `media.py` | 4,684 | Hierarchical data model: `Video`, `Channel`, `Playlist`, `Folder`, `Scheduled`, plus `GenericMedia` / `GenericContainer` / `GenericRemoteContainer` mixins. |
|
||||||
|
| `wizwin.py` | 4,113 | Setup wizard, YouTube subscriptions import wizard, tutorial wizard. |
|
||||||
|
| `options.py` | 2,284 | `OptionsManager`: a 164-field bag of yt-dlp / youtube-dl flags that can be attached to any Video / Channel / Playlist / Folder, plus an `OptionsParser` that compiles them into a yt-dlp CLI invocation. |
|
||||||
|
| `tidy.py` | 1,636 | `TidyManager`: verify-on-disk operation that finds phantom DB records, missing files, orphaned thumbnails. |
|
||||||
|
| `ffmpeg_tartube.py` | 1,303 | `FFmpegManager` + `FFmpegOptionsManager`: post-processing pipeline with named presets. |
|
||||||
|
| `formats.py` | 1,289 | Canonical lists: SponsorBlock categories/actions, audio/video format codes, language codes, livestream message patterns. |
|
||||||
|
| `updates.py` | 1,180 | `UpdateManager`: self-update yt-dlp via pip / system package / git inside Tartube. |
|
||||||
|
| `process.py` | 1,012 | `ProcessManager`: chains FFmpeg invocations after a download completes. |
|
||||||
|
| `info.py` | 671 | `InfoManager`: yt-dlp `--dump-json` + capability probes, version checks. |
|
||||||
|
| `refresh.py` | 617 | `RefreshManager`: re-import disk state into the media registry. |
|
||||||
|
| `dialogue.py` | 522 | Small/common dialog primitives (yes-no, message, OK). |
|
||||||
|
| `classes.py` | small | Misc shared helpers. |
|
||||||
|
| `files.py` | 167 | File-existence checks. |
|
||||||
|
| `xdg_tartube.py` | 183 | XDG base-directory resolution for config/data paths. |
|
||||||
|
|
||||||
|
Auxiliary directories:
|
||||||
|
|
||||||
|
- `locale/` — `gettext` translations (many languages).
|
||||||
|
- `icons/`, `sounds/` — assets.
|
||||||
|
- `nsis/`, `pack/` — Windows installer + per-distro packaging.
|
||||||
|
- `docs/` — manual pages.
|
||||||
|
|
||||||
|
**What we can do:** Our entire codebase is ~9k lines of Rust right now.
|
||||||
|
Tartube's volume comes from (a) the sheer number of UI dialogs (~80) and
|
||||||
|
(b) `OptionsManager` cascading through every code path. Both are
|
||||||
|
mechanical work to match, not architectural blockers.
|
||||||
|
|
||||||
|
## 2. Data model
|
||||||
|
|
||||||
|
Tartube's central insight is a **container hierarchy** where everything
|
||||||
|
nests:
|
||||||
|
|
||||||
|
```
|
||||||
|
Folder ─┬─ Folder (recursive)
|
||||||
|
├─ Channel ── Video
|
||||||
|
└─ Playlist ── Video
|
||||||
|
```
|
||||||
|
|
||||||
|
`Folder` is the freeform organisation layer (user-created and
|
||||||
|
system-created — see §2.1). `Channel` and `Playlist` are "remote
|
||||||
|
containers" with a `source` URL and download semantics. `Video` is a
|
||||||
|
leaf with on-disk file references plus state flags.
|
||||||
|
|
||||||
|
### 2.1 System folders
|
||||||
|
|
||||||
|
Tartube ships with a fixed set of system `Folder` objects users can't
|
||||||
|
delete:
|
||||||
|
|
||||||
|
- **All Videos** — every video across the registry.
|
||||||
|
- **Bookmarks** — videos with `bookmark_flag = True`.
|
||||||
|
- **Favourite Videos** — videos with `fav_flag = True`, inherited from
|
||||||
|
any ancestor.
|
||||||
|
- **Live Videos** — videos with `live_mode > 0`.
|
||||||
|
- **Missing Videos** — videos that existed but are gone from the source.
|
||||||
|
- **New Videos** — downloaded but not yet watched.
|
||||||
|
- **Recent Videos** — last N downloads.
|
||||||
|
- **Waiting Videos** — videos manually queued for later attention.
|
||||||
|
- **Temporary Videos** — one-off Classic-Mode downloads.
|
||||||
|
|
||||||
|
System folders are virtual: they don't own their videos, they're
|
||||||
|
filtered views of the registry by flag. Adding a video to "Bookmarks"
|
||||||
|
sets `bookmark_flag` on the underlying video; removing the bookmark
|
||||||
|
removes the video from the folder.
|
||||||
|
|
||||||
|
**What we can do:** This pattern is *exactly* the activity feed / smart
|
||||||
|
filter mechanism we should adopt. Our `SidebarView::Recent` is one of
|
||||||
|
these, hand-coded. Generalising into a `SmartFolder` enum with a
|
||||||
|
predicate would let us add Bookmarks / Favourites / Waiting / New
|
||||||
|
mechanically. Tartube parity-and-then-some.
|
||||||
|
|
||||||
|
### 2.2 `media.Video` field inventory
|
||||||
|
|
||||||
|
Beyond what we have today, Tartube tracks per video:
|
||||||
|
|
||||||
|
| Field | Purpose | Have it? |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `dbid` | Integer primary key | No (we key by yt-dlp ID string) |
|
||||||
|
| `vid` | Extractor's video ID | Yes (`Video::id`) |
|
||||||
|
| `name` vs `nickname` | Filename vs display title | No — one `title` field |
|
||||||
|
| `natname` | Natural-sort key | No — sorted at query time |
|
||||||
|
| `source` | Original URL | No on `Video`; we keep `source_url` per Channel |
|
||||||
|
| `file_name`, `file_ext` | Separate parts | No — one `PathBuf` |
|
||||||
|
| `file_size` | bytes | Yes |
|
||||||
|
| `upload_time` (unix) | Upload date as seconds | Partial — we have `YYYYMMDD` string |
|
||||||
|
| `receive_time` (unix) | When Tartube downloaded it | We have `mtime_unix`, equivalent |
|
||||||
|
| `duration` (int seconds) | Yes (`duration_secs`) |
|
||||||
|
| `index` | Position in channel/playlist | No |
|
||||||
|
| `author` | Uploader (distinct from channel) | No |
|
||||||
|
| `descrip`, `short` | Full + summary description | We have full via sidecar |
|
||||||
|
| `live_mode` (0/1/2) | Not-live / scheduled / broadcasting | We have `live: bool` at download time only |
|
||||||
|
| `live_debut_flag` | YouTube "premiere" videos | No |
|
||||||
|
| `was_live_flag` | Once-was-live, prevents re-marking | No |
|
||||||
|
| `live_time` | Approximate start time | No |
|
||||||
|
| `live_msg` | Parsed livestream message | No |
|
||||||
|
| `archive_flag` | Skip auto-deletion | No |
|
||||||
|
| `bookmark_flag` | Bookmarked | No |
|
||||||
|
| `fav_flag` | Favourited (cascades from ancestors) | No |
|
||||||
|
| `missing_flag` | Removed from source after download | No (we have a maintenance scan) |
|
||||||
|
| `new_flag` | Downloaded but unwatched | Inverse of our `watched` set |
|
||||||
|
| `waiting_flag` | Manually queued for attention | No |
|
||||||
|
| `block_flag` | Censored / age-restricted | No |
|
||||||
|
| `split_flag` | Clip extracted from a parent video | No |
|
||||||
|
| `orig_parent_obj` | Where the video originally came from | No |
|
||||||
|
| `subs_list` | Available subtitle languages | Yes (`subtitles: Vec<Subtitle>`) |
|
||||||
|
| `stamp_list` | Chapter timestamps (manual or extracted) | Partial — we read chapters from info.json, no manual editing |
|
||||||
|
| `slice_list` | Per-video SponsorBlock data | No — we use `--sponsorblock-mark` at download time |
|
||||||
|
| `comment_list` | Comments (yt-dlp `--write-comments`) | No |
|
||||||
|
| `error_list`, `warning_list` | Last download attempt messages | No persisted; we have rolling Job log only |
|
||||||
|
|
||||||
|
### 2.3 `media.Channel` / `media.Playlist` / `media.Folder`
|
||||||
|
|
||||||
|
Container fields (shared via `GenericContainer`):
|
||||||
|
|
||||||
|
- `dbid`, `name`, `nickname`, `natname`, `parent_obj`, `child_list`.
|
||||||
|
- `options_obj` — per-container `OptionsManager` override (§3).
|
||||||
|
- Counters: `vid_count`, `bookmark_count`, `dl_count`, `fav_count`,
|
||||||
|
`live_count`, `missing_count`, `new_count`, `waiting_count`.
|
||||||
|
- `fav_flag` — cascades to all descendants.
|
||||||
|
- `error_list`, `warning_list`.
|
||||||
|
- `external_dir` — optional override storage path.
|
||||||
|
- `master_dbid` / `slave_dbid_list` — alias resolution (a video uploaded
|
||||||
|
to two channels with different IDs can be deduplicated by pointing one
|
||||||
|
to the other).
|
||||||
|
|
||||||
|
Channels & Playlists additionally have:
|
||||||
|
|
||||||
|
- `source` (URL).
|
||||||
|
- `rss` — RSS feed URL (for fast new-video discovery).
|
||||||
|
- `playlist_id_dict` — set of playlist IDs already imported.
|
||||||
|
- `dl_no_db_flag` — download to disk but skip the database.
|
||||||
|
- `dl_sim_flag` — always simulate (check-only, never actually fetch).
|
||||||
|
- `dl_disable_flag` — skip during operations.
|
||||||
|
|
||||||
|
### 2.4 Storage
|
||||||
|
|
||||||
|
- **`tartube.db`** — Python `pickle` of the entire media registry, not
|
||||||
|
SQLite. Atomic save-on-modify with periodic backups (configurable
|
||||||
|
`db_backup_mode = 'always' | 'every_session' | 'daily' | 'never'`).
|
||||||
|
- **Per-channel folder layout**:
|
||||||
|
```
|
||||||
|
<data_dir>/
|
||||||
|
<channel_name>/
|
||||||
|
<video_name>.mp4
|
||||||
|
<video_name>.info.json
|
||||||
|
<video_name>.description
|
||||||
|
<video_name>.<ext>.<lang>.vtt
|
||||||
|
.ytdl-archive (yt-dlp's downloaded-IDs record)
|
||||||
|
...
|
||||||
|
```
|
||||||
|
- **External directories** — a channel can be redirected to a separate
|
||||||
|
drive via `external_dir`.
|
||||||
|
|
||||||
|
**What we can do:** Pickle is a misfeature — schema migrations are
|
||||||
|
nightmarish and a corrupt pickle locks out the whole library. Our
|
||||||
|
SQLite-backed approach is strictly better. Don't adopt the storage
|
||||||
|
model; do adopt the per-channel layout (which matches ours already).
|
||||||
|
|
||||||
|
## 3. `OptionsManager` — the headline feature
|
||||||
|
|
||||||
|
`options.OptionsManager` is the single biggest gap between us and
|
||||||
|
Tartube. It holds **164 distinct yt-dlp / youtube-dl flags** and can
|
||||||
|
be attached to:
|
||||||
|
|
||||||
|
- A specific `Video`.
|
||||||
|
- A `Channel` / `Playlist` / `Folder`.
|
||||||
|
- The application globally (the "default" manager).
|
||||||
|
|
||||||
|
Resolution at download time: `Video.options_obj` → `parent.options_obj`
|
||||||
|
→ ancestor chain → app global. First non-`None` wins. A user can
|
||||||
|
create *named* OptionsManager instances and apply the same one to many
|
||||||
|
channels.
|
||||||
|
|
||||||
|
### 3.1 Canonical option groups
|
||||||
|
|
||||||
|
From the docstring at `options.py:69`:
|
||||||
|
|
||||||
|
- **Behaviour**: ignore_errors, abort_on_error, live_from_start, wait_for_video_min.
|
||||||
|
- **Network**: proxy, socket_timeout, source_address, force_ipv4/6,
|
||||||
|
geo_verification_proxy, geo_bypass*, geo_bypass_country, geo_bypass_ip_block.
|
||||||
|
- **Playlist selection**: playlist_start, playlist_end, playlist_items,
|
||||||
|
max_downloads, playlist_reverse, playlist_random,
|
||||||
|
skip_playlist_after_errors, break_on_existing, break_on_reject.
|
||||||
|
- **Filters**: min_filesize, max_filesize, date, date_before, date_after,
|
||||||
|
min_views, max_views, match_filter, age_limit, include_ads,
|
||||||
|
match_title_list, reject_title_list.
|
||||||
|
- **Download**: limit_rate, retries, abort_on_unavailable_fragment,
|
||||||
|
native_hls, hls_prefer_ffmpeg, external_downloader, external_arg_string,
|
||||||
|
concurrent_fragments, throttled_rate.
|
||||||
|
- **Filesystem**: restrict_filenames, nomtime, write_description,
|
||||||
|
write_info, write_annotations, cookies_path, write_thumbnail,
|
||||||
|
force_encoding, output_format, output_template, output_format_list,
|
||||||
|
output_path_list, windows_filenames, trim_filenames, no_overwrites,
|
||||||
|
force_overwrites.
|
||||||
|
- **Auth**: username, password, two_factor, net_rc, video_password,
|
||||||
|
ap_mso, ap_username, ap_password.
|
||||||
|
- **Anti-bot**: no_check_certificate, prefer_insecure, user_agent,
|
||||||
|
referer, min_sleep_interval, max_sleep_interval.
|
||||||
|
- **Format**: video_format, all_formats, prefer_free_formats, yt_skip_dash,
|
||||||
|
merge_output_format, video_format_list, video_format_mode.
|
||||||
|
- **Subtitles**: write_subs, write_auto_subs, write_all_subs, subs_format,
|
||||||
|
subs_lang, subs_lang_list.
|
||||||
|
- **Post-processing**: extract_audio, audio_format, audio_quality,
|
||||||
|
recode_video, pp_args, keep_video, embed_subs, embed_thumbnail,
|
||||||
|
add_metadata, fixup_policy, prefer_avconv, prefer_ffmpeg.
|
||||||
|
- **Cookies**: no_cookies, cookies_from_browser, no_cookies_from_browser.
|
||||||
|
- **Output sidecars**: write_link, write_url_link, write_webloc_link,
|
||||||
|
write_desktop_link.
|
||||||
|
- **Tartube-specific**: keep_description / keep_info / keep_thumbnail /
|
||||||
|
keep_annotations (move sidecars vs delete), sim_keep_* (same in
|
||||||
|
simulate mode), move_description (move out of data dir after
|
||||||
|
download), extra_cmd_string (raw passthrough), direct_cmd_flag /
|
||||||
|
direct_url_flag (escape hatches), fetch_formats_cmd_string,
|
||||||
|
fetch_subtitles_cmd_string, downloader_config (write a yt-dlp config
|
||||||
|
file), check_fetch_comments / dl_fetch_comments /
|
||||||
|
store_comments_in_db, extractor_args_list.
|
||||||
|
|
||||||
|
### 3.2 `OptionsParser`
|
||||||
|
|
||||||
|
A second class that walks the canonical option list (a registry of
|
||||||
|
`OptionHolder` entries) and emits a yt-dlp CLI: `--retries 30
|
||||||
|
--match-filter "duration > 60" -o "<template>" --cookies <path>` and
|
||||||
|
so on. Each `OptionHolder` knows its option name, its CLI switch,
|
||||||
|
default value, and `requirement_list` (other options that must be set).
|
||||||
|
|
||||||
|
**What we can do:** This is Phase 1.1 of our ROADMAP. The concrete
|
||||||
|
design:
|
||||||
|
|
||||||
|
1. Add a `download_options` table to SQLite: `(id, name, parent_id,
|
||||||
|
json_blob)`.
|
||||||
|
2. Attach to channels via `channels` table additions: `options_id`.
|
||||||
|
3. New `DownloadOptions` Rust struct with serde — start with maybe 20
|
||||||
|
most-used flags, not all 164. Skip the ones we don't need
|
||||||
|
(`force_ipv4` etc.).
|
||||||
|
4. Resolver: `Video.options` → `Channel.options` → app default.
|
||||||
|
5. Per-channel "Edit options…" dialog mirrors the structure of
|
||||||
|
`OptionsEditWin`.
|
||||||
|
|
||||||
|
## 4. Operations model
|
||||||
|
|
||||||
|
Tartube has **seven distinct operations**, each its own threaded class.
|
||||||
|
Only one operation can run at a time except for live-stream monitoring,
|
||||||
|
which runs concurrently.
|
||||||
|
|
||||||
|
| Operation | Class | Purpose |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Download | `DownloadManager` | Actually fetch videos via yt-dlp |
|
||||||
|
| Update | `UpdateManager` | Self-update yt-dlp (pip / system / git) |
|
||||||
|
| Refresh | `RefreshManager` | Re-import disk state into the media registry |
|
||||||
|
| Tidy | `TidyManager` | Verify files-on-disk vs DB, clean phantoms |
|
||||||
|
| Info | `InfoManager` | yt-dlp `--dump-json` probes, version checks, test runs |
|
||||||
|
| Process | `ProcessManager` | Post-download FFmpeg pipeline |
|
||||||
|
| Livestream | `StreamManager` | Background polling for live channels |
|
||||||
|
|
||||||
|
### 4.1 `DownloadManager`
|
||||||
|
|
||||||
|
Threading model:
|
||||||
|
|
||||||
|
- `DownloadManager` is a `threading.Thread`. It owns a worker pool of
|
||||||
|
`DownloadWorker` threads (configurable max, default 2).
|
||||||
|
- The `DownloadList` is a queue of `DownloadItem` objects. Each item
|
||||||
|
references a `media.Video` / `Channel` / `Playlist` plus an
|
||||||
|
`operation_type` (sim / real / classic_sim / classic_real /
|
||||||
|
custom_sim / custom_real).
|
||||||
|
- A worker pulls the next item, instantiates a `VideoDownloader` /
|
||||||
|
`ClipDownloader` / `StreamDownloader`, runs it, reports back.
|
||||||
|
- Bypass workers handle livestreams without consuming a normal slot.
|
||||||
|
- Progress is reported via a `data_callback` that updates the Progress
|
||||||
|
tab in real time.
|
||||||
|
|
||||||
|
### 4.2 `VideoDownloader` vs `ClipDownloader` vs `StreamDownloader`
|
||||||
|
|
||||||
|
- **`VideoDownloader`** — the standard case. Builds a yt-dlp CLI from
|
||||||
|
the resolved `OptionsManager`, runs it, parses stdout/stderr line by
|
||||||
|
line, updates the `media.Video` record on completion.
|
||||||
|
- **`ClipDownloader`** — uses yt-dlp's `--download-sections` /
|
||||||
|
`--postprocessor-args "ffmpeg:-ss …"` to download just one timestamp
|
||||||
|
range. Powers the "Edit timestamps" → "Download clip" UI.
|
||||||
|
- **`StreamDownloader`** — wraps `yt-dlp --live-from-start` with
|
||||||
|
Tartube-specific monitoring, restart-on-disconnect, and a max-duration
|
||||||
|
cap. Used by `StreamManager`'s livestream auto-record.
|
||||||
|
|
||||||
|
### 4.3 `Tidy`, `Refresh`, `Info`
|
||||||
|
|
||||||
|
- **Tidy** — like our maintenance scan but more aggressive: finds
|
||||||
|
duplicate dbids, orphaned files, channels whose folder vanished,
|
||||||
|
videos that exist on disk but not in the DB.
|
||||||
|
- **Refresh** — opposite direction: walks the disk and imports any
|
||||||
|
unrecognised file into the DB.
|
||||||
|
- **Info** — runs yt-dlp `--dump-json` on a URL to fetch metadata
|
||||||
|
without downloading; also `--test` mode that downloads only a
|
||||||
|
fragment to verify settings work.
|
||||||
|
|
||||||
|
### 4.4 Update operation
|
||||||
|
|
||||||
|
Three backends, user-selectable:
|
||||||
|
|
||||||
|
- pip (`pip install -U yt-dlp`)
|
||||||
|
- system package (`apt install --reinstall yt-dlp`, etc.)
|
||||||
|
- git pull on a local clone
|
||||||
|
|
||||||
|
**What we can do:** Our bundled-venv install is conceptually the
|
||||||
|
"pip" path. We don't expose a "git pull from upstream master" mode but
|
||||||
|
arguably that's overkill for a single-user tool.
|
||||||
|
|
||||||
|
## 5. Custom downloads
|
||||||
|
|
||||||
|
`CustomDLManager` adds a second axis of configuration on top of
|
||||||
|
`OptionsManager`. A custom download is a *recipe*:
|
||||||
|
|
||||||
|
- Which sources to use (channels / playlists / folders / specific
|
||||||
|
videos).
|
||||||
|
- Whether to download or just check.
|
||||||
|
- Whether to delete originals after a successful post-process.
|
||||||
|
- A specific `OptionsManager`.
|
||||||
|
- A scheduled time (via `Scheduled`).
|
||||||
|
|
||||||
|
Users define multiple named custom-DL recipes and trigger them
|
||||||
|
manually from the Drag-and-Drop tab or via schedule.
|
||||||
|
|
||||||
|
**What we can do:** Our "Music mode" + "Live mode" + "Twitch clips
|
||||||
|
only" toggles are early hints of this. The principled version is
|
||||||
|
*named profiles* the user can construct and reuse.
|
||||||
|
|
||||||
|
## 6. UI surfaces
|
||||||
|
|
||||||
|
`MainWin` is one Gtk window with five primary tabs:
|
||||||
|
|
||||||
|
1. **Videos** — the main library browser. Sidebar = container tree;
|
||||||
|
main area = "Video Catalogue" (the video card grid). Three render
|
||||||
|
modes: Simple (list), Complex (cards with metadata), Grid
|
||||||
|
(thumbnail wall). Filters (by name/author/desc/comment) + sort.
|
||||||
|
2. **Progress** — live download status: per-job rows with status,
|
||||||
|
percent, ETA, current file. Like our jobs list.
|
||||||
|
3. **Classic Mode** — one-off downloads. Paste a URL, pick destination
|
||||||
|
folder, pick format, queue. Bypasses the database entirely.
|
||||||
|
4. **Drag and Drop** — "drop zones" the user creates, each with a
|
||||||
|
pre-configured `OptionsManager`. Dropping a URL onto a zone
|
||||||
|
downloads with that zone's settings.
|
||||||
|
5. **Errors / Warnings** — accumulated diagnostic messages.
|
||||||
|
|
||||||
|
Around the central tabs:
|
||||||
|
|
||||||
|
- Menu bar: Media (add channel / playlist / folder / video, bulk add,
|
||||||
|
insert video, import YouTube subscriptions…), Operations (download,
|
||||||
|
check, tidy, refresh, info, update), Options (edit general /
|
||||||
|
custom-DL / FFmpeg / preferences), Help (about, manual, tutorial).
|
||||||
|
- Toolbar: hide / squeeze, custom icons, custom DL button.
|
||||||
|
- Status icon (system tray): tray menu, restore from tray, minimize to
|
||||||
|
tray.
|
||||||
|
|
||||||
|
### 6.1 Dialogs (~80 of them)
|
||||||
|
|
||||||
|
A representative sample:
|
||||||
|
|
||||||
|
- **AddChannelDialogue / AddPlaylistDialogue / AddFolderDialogue** —
|
||||||
|
add a remote container with name + URL + parent folder.
|
||||||
|
- **AddBulkDialogue** — paste many URLs, autodetect type per URL.
|
||||||
|
- **AddStampDialogue** — manually add chapter timestamps.
|
||||||
|
- **AddVideoDialogue** — add a single video URL to an existing channel.
|
||||||
|
- **AddDropZoneDialogue** — create a Drag-and-Drop zone.
|
||||||
|
- **ApplyOptionsDialogue** — pick an `OptionsManager` to apply to one
|
||||||
|
or more containers.
|
||||||
|
- **CalendarDialogue** — date picker for filters.
|
||||||
|
- **ChangeThemeDialogue** — switch GTK theme.
|
||||||
|
- **CreateProfileDialogue** — group containers into a "profile" you can
|
||||||
|
download all at once.
|
||||||
|
- **DeleteContainerDialogue / DeleteVideoDialogue / DeleteDropZoneDialogue**
|
||||||
|
— confirm-before-delete with the usual checkboxes.
|
||||||
|
- **DuplicateVideoDialogue** — what to do when the same video ID
|
||||||
|
appears in two channels.
|
||||||
|
- **ExportDialogue / ImportDialogue** — DB export/import as JSON.
|
||||||
|
- **ExtractorCodeDialogue** — manual yt-dlp extractor selection.
|
||||||
|
- **FormatsSubsDialogue** — pick subtitle formats for a download.
|
||||||
|
- **InsertVideoDialogue** — insert a specific video at a position in a
|
||||||
|
playlist.
|
||||||
|
- **MountDriveDialogue** — handle external-drive containers when the
|
||||||
|
drive isn't mounted.
|
||||||
|
|
||||||
|
Plus per-object edit windows in `config.py`:
|
||||||
|
|
||||||
|
- `VideoEditWin` — edit a single video's metadata, flags, stamps,
|
||||||
|
slices.
|
||||||
|
- `ChannelPlaylistEditWin` — channel/playlist editor.
|
||||||
|
- `FolderEditWin` — folder editor.
|
||||||
|
- `OptionsEditWin` — a giant tabbed editor for an `OptionsManager`.
|
||||||
|
- `CustomDLEditWin` — recipe editor.
|
||||||
|
- `FFmpegOptionsEditWin` — preset editor for the post-process pipeline.
|
||||||
|
- `ScheduledEditWin` — schedule entry editor.
|
||||||
|
- `SystemPrefWin` — application-wide preferences (the equivalent of
|
||||||
|
our Settings modal but with ~30 tabs).
|
||||||
|
|
||||||
|
### 6.2 Wizards
|
||||||
|
|
||||||
|
- **SetupWizWin** — first-run wizard: pick data dir, pick downloader,
|
||||||
|
optionally install yt-dlp / FFmpeg, optionally enable the database.
|
||||||
|
- **ImportYTWizWin** — import a YouTube subscriptions OPML/JSON file as
|
||||||
|
channels.
|
||||||
|
- **TutorialWizWin** — embedded tutorial.
|
||||||
|
|
||||||
|
**What we can do:** Our `Settings` modal covers maybe 15% of what
|
||||||
|
`SystemPrefWin` handles. Some of those — proxy config, retry tuning,
|
||||||
|
sleep intervals — are reasonable to add. Others (Adobe Pass auth, geo
|
||||||
|
bypass, restrict filenames) are corner cases that bloat the surface
|
||||||
|
without serving a clear use case. Be selective.
|
||||||
|
|
||||||
|
## 7. Configuration / preferences
|
||||||
|
|
||||||
|
Tartube's `SystemPrefWin` covers (each is a tab):
|
||||||
|
|
||||||
|
- General
|
||||||
|
- Files (data dir, external dirs, backups)
|
||||||
|
- Downloads (worker count, retry limits, bandwidth)
|
||||||
|
- Filters (default catalogue filters)
|
||||||
|
- Display (themes, drag formats, status icon)
|
||||||
|
- Operations (sim vs real defaults, error visibility)
|
||||||
|
- yt-dlp (path to binary, update mode, extra args)
|
||||||
|
- FFmpeg (post-process presets, codec defaults)
|
||||||
|
- Livestreams (polling interval, max duration)
|
||||||
|
- Notifications (system, dialog, sound)
|
||||||
|
- Scheduling (default times, day-of-week toggles)
|
||||||
|
- Profiles (named container groups)
|
||||||
|
- Cookies (path, browser, refresh policy)
|
||||||
|
- Anti-bot (sleep intervals, user agents, impersonation hints —
|
||||||
|
available since yt-dlp added curl_cffi)
|
||||||
|
- SponsorBlock (categories, actions, manual editing toggle)
|
||||||
|
- Comments (fetch / store / display)
|
||||||
|
- Advanced (locks, debug toggles, raw command-line passthrough)
|
||||||
|
|
||||||
|
The 791 boolean flags in `mainapp.py` cover all of these plus their
|
||||||
|
"…also show this…" / "…also do this in classic mode…" variants.
|
||||||
|
|
||||||
|
**What we can do:** A handful of these tabs map to features we
|
||||||
|
literally don't have (FFmpeg presets, comments, profiles, scheduling
|
||||||
|
calendar). They're roadmap items, not config surface. The actual
|
||||||
|
*configuration surface* of yt-offline is currently:
|
||||||
|
|
||||||
|
- backup dir, max concurrent, bundled-or-system yt-dlp
|
||||||
|
- player command, browser (for `--cookies-from-browser`)
|
||||||
|
- UI theme
|
||||||
|
- web port, bind interface, transcode, source URL
|
||||||
|
- scheduler enabled + interval hours
|
||||||
|
- Plex library path
|
||||||
|
- cookies (paste/file)
|
||||||
|
- password (Argon2)
|
||||||
|
|
||||||
|
Adding ~5–8 fields (proxy, retry overrides, sleep interval, default
|
||||||
|
options manager) would close most of the practical gap. Adding 50 more
|
||||||
|
matches Tartube but rapidly costs more than it earns.
|
||||||
|
|
||||||
|
## 8. SponsorBlock + Stamp lists
|
||||||
|
|
||||||
|
Tartube goes further than yt-dlp's built-in SponsorBlock support:
|
||||||
|
|
||||||
|
- `media.Video.slice_list` stores per-video segments with `(category,
|
||||||
|
action, start_time, stop_time)`.
|
||||||
|
- The user can manually add slices via a UI editor.
|
||||||
|
- The slice list survives re-downloads — Tartube re-applies them.
|
||||||
|
- Slices can be set to "skip" (cut from output) instead of just "mark"
|
||||||
|
(chapter marker).
|
||||||
|
|
||||||
|
`stamp_list` is the same idea for chapters: extracted from
|
||||||
|
description/metadata at download time, editable by the user, used by
|
||||||
|
`ClipDownloader` to slice the source video into named clips.
|
||||||
|
|
||||||
|
**What we can do:** Our SponsorBlock support is "pass `--sponsorblock-mark
|
||||||
|
all` to yt-dlp and hope". Adopting Tartube's persistence model would
|
||||||
|
let users:
|
||||||
|
- Manually mark intros / outros / sponsor reads on any video.
|
||||||
|
- Generate clips by chapter (download a 30-minute video, save 5 clips
|
||||||
|
named after their chapter titles).
|
||||||
|
- Hide already-cut segments from the player.
|
||||||
|
|
||||||
|
Reasonable Phase 1 add-on after per-channel options ships.
|
||||||
|
|
||||||
|
## 9. Notable corners
|
||||||
|
|
||||||
|
- **Master/slave dbid** — alias resolution for re-uploads. A video
|
||||||
|
uploaded to two channels with different IDs can be marked as a
|
||||||
|
slave of another, so the dedupe scan treats them as the same.
|
||||||
|
- **Profiles** — named groups of `Channel` / `Playlist` / `Folder`
|
||||||
|
references. The user can "Download Profile 'Daily'" and get every
|
||||||
|
member checked in one batch.
|
||||||
|
- **Newbie dialog** — pop-up that surfaces useful tips for new users.
|
||||||
|
- **RSS feed discovery** — `media.Channel.rss` lets Tartube use a
|
||||||
|
channel's RSS feed for fast new-video discovery instead of doing a
|
||||||
|
full yt-dlp playlist scan.
|
||||||
|
- **Livestream message parsing** — `ttutils.extract_livestream_data()`
|
||||||
|
parses YouTube's "Scheduled for…" / "Premieres in 3 days" strings
|
||||||
|
into structured fields used for sorting + display.
|
||||||
|
- **External directories** — a single library can span multiple drives
|
||||||
|
by setting `media.Channel.external_dir` per-channel.
|
||||||
|
- **Block flag** — videos that are age-restricted / geo-blocked /
|
||||||
|
censored. Tartube tracks them separately so the UI shows them with
|
||||||
|
a different colour and the download skips them.
|
||||||
|
- **Locale** — full `gettext` i18n with ~10 translations. We have
|
||||||
|
none.
|
||||||
|
|
||||||
|
## 10. Things Tartube does better than us
|
||||||
|
|
||||||
|
Items where we *currently lose* and should adopt:
|
||||||
|
|
||||||
|
1. **Per-target download options with cascade resolution** (`OptionsManager`)
|
||||||
|
— Phase 1.1 of our roadmap. Largest gap.
|
||||||
|
2. **Folder hierarchy** — Phase 1.2.
|
||||||
|
3. **System folders as smart filters** — Bookmarks, Favourites, Waiting,
|
||||||
|
New, Live, Missing. Generalises our `SidebarView::Recent`.
|
||||||
|
4. **Multiple per-video state flags** — bookmark / favourite / archive /
|
||||||
|
missing / waiting / block / new. Currently we only track watched.
|
||||||
|
5. **Master / slave dedup** — alias one upload to another.
|
||||||
|
6. **Manual chapter / clip editing UI** — `stamp_list` + `ClipDownloader`.
|
||||||
|
7. **Manual SponsorBlock editing + persistence** — `slice_list`.
|
||||||
|
8. **Comments capture and viewer** — `--write-comments` + `comment_list`.
|
||||||
|
9. **Profiles** — named container groups for batch operations.
|
||||||
|
10. **Tidy + Refresh operations** — full bidirectional sync between
|
||||||
|
registry and disk. Our maintenance scan covers only half.
|
||||||
|
11. **Setup wizard** — first-run handholding for the data dir / yt-dlp
|
||||||
|
/ FFmpeg.
|
||||||
|
12. **i18n / `gettext`** — every locale Tartube has, we don't.
|
||||||
|
13. **Per-distro installer ecosystem** — .deb, .rpm, MSI, .dmg,
|
||||||
|
Chocolatey, AUR. We ship a PKGBUILD.
|
||||||
|
14. **RSS-driven new-video discovery** — quicker than a full yt-dlp
|
||||||
|
scan.
|
||||||
|
15. **`extra_cmd_string` / passthrough** — escape hatch for any yt-dlp
|
||||||
|
flag we haven't surfaced. We have nothing.
|
||||||
|
|
||||||
|
## 11. Things we do better than Tartube
|
||||||
|
|
||||||
|
For the record (to keep momentum honest):
|
||||||
|
|
||||||
|
1. **Single static binary** vs Python + GTK + dozens of pip deps.
|
||||||
|
2. **Web UI accessible from any device** vs desktop-only GTK window.
|
||||||
|
3. **Modern security stack** — Argon2 sessions, CSP, rate-limit, body
|
||||||
|
limit, ETag — vs none.
|
||||||
|
4. **First-class Plex export** with NFO sidecars and show folders.
|
||||||
|
5. **Bundled curl_cffi venv install** — Tartube users do this manually
|
||||||
|
per platform.
|
||||||
|
6. **Real multi-platform routing** — separate `tiktok/`, `twitch/`,
|
||||||
|
`vimeo/` etc. with `.source-url` sidecars. Tartube treats every
|
||||||
|
source identically.
|
||||||
|
7. **Themes** — 10 of ours vs default GTK plus a single optional dark
|
||||||
|
theme.
|
||||||
|
8. **Performance** — Rust parallel scan, gzip + ETag on
|
||||||
|
`/api/library`, adaptive polling, DB connection pool. Tartube's
|
||||||
|
pickle load is the bottleneck for large libraries.
|
||||||
|
9. **SHA-256-verified bundled-binary install**.
|
||||||
|
10. **Activity feed / Recent additions sidebar entry** — Tartube has
|
||||||
|
a Recent Videos folder but no per-week activity histogram.
|
||||||
|
|
||||||
|
## 12. Specification for "Tartube parity" milestone
|
||||||
|
|
||||||
|
Working definition: a yt-offline release is "at Tartube parity" when
|
||||||
|
the following are all true.
|
||||||
|
|
||||||
|
- [ ] **Per-target download options** with cascade resolution and a
|
||||||
|
per-channel editor.
|
||||||
|
- [ ] **Folder hierarchy** with N-level nesting, drag-to-reparent, and
|
||||||
|
per-folder default options.
|
||||||
|
- [ ] **Smart folders** for Bookmarks / Favourites / Waiting / New /
|
||||||
|
Live / Missing, on top of our existing Recent + Continue
|
||||||
|
Watching.
|
||||||
|
- [ ] **Per-video state flags** persisted (extending the SQLite
|
||||||
|
schema): `bookmark`, `favourite`, `archive`, `waiting`,
|
||||||
|
`block`, plus a `missing` derived flag from the maintenance
|
||||||
|
scan.
|
||||||
|
- [ ] **Profiles** — saved named groups of channels with a "Download
|
||||||
|
this profile" action.
|
||||||
|
- [ ] **Tidy + Refresh** unified into a single bidirectional sync
|
||||||
|
operation in the maintenance window.
|
||||||
|
- [ ] **Comments capture and viewer** — opt-in `--write-comments`
|
||||||
|
flag; new viewer in the video player modal.
|
||||||
|
- [ ] **Manual chapter / SponsorBlock editing UI** — edit / persist /
|
||||||
|
re-apply on re-download.
|
||||||
|
- [ ] **Setup wizard** — first-run flow for empty installs that
|
||||||
|
handles data dir, yt-dlp install, optional FFmpeg, optional
|
||||||
|
Plex.
|
||||||
|
- [ ] **Per-distro release artifacts** — .deb, .rpm, MSI, .app/.dmg,
|
||||||
|
AUR, Chocolatey or winget.
|
||||||
|
- [ ] **`extra_args` passthrough** field in download options for any
|
||||||
|
flag we don't surface natively.
|
||||||
|
- [ ] **i18n scaffolding** — even one alt locale to prove the build.
|
||||||
|
|
||||||
|
When all twelve land, the comparison table at the top of `ROADMAP.md`
|
||||||
|
goes from 8-ahead-8-behind to 8-ahead-0-behind. After that, the
|
||||||
|
"surpass" phase consists of leaning on our architectural advantages:
|
||||||
|
WebSocket progress, Android client, federation, perceptual-hash dedup,
|
||||||
|
plugins.
|
||||||
|
|
||||||
|
## 13. References
|
||||||
|
|
||||||
|
- Source: <https://github.com/axcore/tartube>
|
||||||
|
- Local clone for this analysis: `/tmp/tartube` (axcore/tartube
|
||||||
|
v2.5.231).
|
||||||
|
- Modules read in full: `media.py` (data model), `options.py` (option
|
||||||
|
registry), `downloads.py` (operations + workers), `mainapp.py`
|
||||||
|
(config surface).
|
||||||
|
- Modules summarised: every other `.py` file under `tartube/`.
|
||||||
|
- Cross-referenced with our `ROADMAP.md` Phase 1 items.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue