diff --git a/ROADMAP.md b/ROADMAP.md index ace3d7f..10c7be6 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -4,6 +4,11 @@ **Surpass [Tartube](https://github.com/axcore/tartube) in every dimension.** +A structured analysis of Tartube's codebase, data model, operations, and +configuration surface lives at [`docs/tartube-spec.md`](docs/tartube-spec.md). +It enumerates the exact features we need to match — every Phase 1 item +below traces back to a specific Tartube subsystem. + Tartube is the mature open-source yt-dlp GUI and the obvious benchmark for a project in this space. yt-offline has architectural advantages Tartube can't catch up on quickly (Rust + axum + a real web UI + bundled toolchain + a diff --git a/docs/tartube-spec.md b/docs/tartube-spec.md new file mode 100644 index 0000000..b7259cc --- /dev/null +++ b/docs/tartube-spec.md @@ -0,0 +1,617 @@ +# Tartube Specification + +A structured analysis of [Tartube v2.5.231](https://github.com/axcore/tartube) +(2026-05-24), our project's primary feature benchmark. Written to give +yt-offline a concrete target to work toward — every section ends with a +"What we can do" note about whether to adopt, adapt, or skip. + +Tartube codebase: ~146,000 lines of Python 3 across 19 modules under +`tartube/`, plus locale, icons, sounds, screenshots. GTK 3 frontend. +GPL-3-or-later. Single-user desktop application. + +## 1. Module map + +| File | Lines | Responsibility | +| --- | --- | --- | +| `mainapp.py` | 30,565 | The `TartubeApp` Gtk.Application — owns 1,601 instance variables / 791 boolean toggles, every config field, every operation entry point. | +| `mainwin.py` | 42,074 | `MainWin` Gtk window + ~80 specialised `Gtk.Dialog` subclasses + the Video Catalogue widgets. | +| `config.py` | 36,659 | Preferences and per-object editor windows (per-channel options, FFmpeg options, scheduled actions, custom-download recipes). | +| `downloads.py` | 12,188 | Download orchestration: `DownloadManager` (worker pool), `DownloadList` (queue), `DownloadWorker`, three downloader strategies (`VideoDownloader`, `ClipDownloader`, `StreamDownloader`), JSON fetchers. | +| `ttutils.py` | 5,015 | Cross-cutting helpers: path manipulation, livestream-message parsing, locale, sorting. | +| `media.py` | 4,684 | Hierarchical data model: `Video`, `Channel`, `Playlist`, `Folder`, `Scheduled`, plus `GenericMedia` / `GenericContainer` / `GenericRemoteContainer` mixins. | +| `wizwin.py` | 4,113 | Setup wizard, YouTube subscriptions import wizard, tutorial wizard. | +| `options.py` | 2,284 | `OptionsManager`: a 164-field bag of yt-dlp / youtube-dl flags that can be attached to any Video / Channel / Playlist / Folder, plus an `OptionsParser` that compiles them into a yt-dlp CLI invocation. | +| `tidy.py` | 1,636 | `TidyManager`: verify-on-disk operation that finds phantom DB records, missing files, orphaned thumbnails. | +| `ffmpeg_tartube.py` | 1,303 | `FFmpegManager` + `FFmpegOptionsManager`: post-processing pipeline with named presets. | +| `formats.py` | 1,289 | Canonical lists: SponsorBlock categories/actions, audio/video format codes, language codes, livestream message patterns. | +| `updates.py` | 1,180 | `UpdateManager`: self-update yt-dlp via pip / system package / git inside Tartube. | +| `process.py` | 1,012 | `ProcessManager`: chains FFmpeg invocations after a download completes. | +| `info.py` | 671 | `InfoManager`: yt-dlp `--dump-json` + capability probes, version checks. | +| `refresh.py` | 617 | `RefreshManager`: re-import disk state into the media registry. | +| `dialogue.py` | 522 | Small/common dialog primitives (yes-no, message, OK). | +| `classes.py` | small | Misc shared helpers. | +| `files.py` | 167 | File-existence checks. | +| `xdg_tartube.py` | 183 | XDG base-directory resolution for config/data paths. | + +Auxiliary directories: + +- `locale/` — `gettext` translations (many languages). +- `icons/`, `sounds/` — assets. +- `nsis/`, `pack/` — Windows installer + per-distro packaging. +- `docs/` — manual pages. + +**What we can do:** Our entire codebase is ~9k lines of Rust right now. +Tartube's volume comes from (a) the sheer number of UI dialogs (~80) and +(b) `OptionsManager` cascading through every code path. Both are +mechanical work to match, not architectural blockers. + +## 2. Data model + +Tartube's central insight is a **container hierarchy** where everything +nests: + +``` +Folder ─┬─ Folder (recursive) + ├─ Channel ── Video + └─ Playlist ── Video +``` + +`Folder` is the freeform organisation layer (user-created and +system-created — see §2.1). `Channel` and `Playlist` are "remote +containers" with a `source` URL and download semantics. `Video` is a +leaf with on-disk file references plus state flags. + +### 2.1 System folders + +Tartube ships with a fixed set of system `Folder` objects users can't +delete: + +- **All Videos** — every video across the registry. +- **Bookmarks** — videos with `bookmark_flag = True`. +- **Favourite Videos** — videos with `fav_flag = True`, inherited from + any ancestor. +- **Live Videos** — videos with `live_mode > 0`. +- **Missing Videos** — videos that existed but are gone from the source. +- **New Videos** — downloaded but not yet watched. +- **Recent Videos** — last N downloads. +- **Waiting Videos** — videos manually queued for later attention. +- **Temporary Videos** — one-off Classic-Mode downloads. + +System folders are virtual: they don't own their videos, they're +filtered views of the registry by flag. Adding a video to "Bookmarks" +sets `bookmark_flag` on the underlying video; removing the bookmark +removes the video from the folder. + +**What we can do:** This pattern is *exactly* the activity feed / smart +filter mechanism we should adopt. Our `SidebarView::Recent` is one of +these, hand-coded. Generalising into a `SmartFolder` enum with a +predicate would let us add Bookmarks / Favourites / Waiting / New +mechanically. Tartube parity-and-then-some. + +### 2.2 `media.Video` field inventory + +Beyond what we have today, Tartube tracks per video: + +| Field | Purpose | Have it? | +| --- | --- | --- | +| `dbid` | Integer primary key | No (we key by yt-dlp ID string) | +| `vid` | Extractor's video ID | Yes (`Video::id`) | +| `name` vs `nickname` | Filename vs display title | No — one `title` field | +| `natname` | Natural-sort key | No — sorted at query time | +| `source` | Original URL | No on `Video`; we keep `source_url` per Channel | +| `file_name`, `file_ext` | Separate parts | No — one `PathBuf` | +| `file_size` | bytes | Yes | +| `upload_time` (unix) | Upload date as seconds | Partial — we have `YYYYMMDD` string | +| `receive_time` (unix) | When Tartube downloaded it | We have `mtime_unix`, equivalent | +| `duration` (int seconds) | Yes (`duration_secs`) | +| `index` | Position in channel/playlist | No | +| `author` | Uploader (distinct from channel) | No | +| `descrip`, `short` | Full + summary description | We have full via sidecar | +| `live_mode` (0/1/2) | Not-live / scheduled / broadcasting | We have `live: bool` at download time only | +| `live_debut_flag` | YouTube "premiere" videos | No | +| `was_live_flag` | Once-was-live, prevents re-marking | No | +| `live_time` | Approximate start time | No | +| `live_msg` | Parsed livestream message | No | +| `archive_flag` | Skip auto-deletion | No | +| `bookmark_flag` | Bookmarked | No | +| `fav_flag` | Favourited (cascades from ancestors) | No | +| `missing_flag` | Removed from source after download | No (we have a maintenance scan) | +| `new_flag` | Downloaded but unwatched | Inverse of our `watched` set | +| `waiting_flag` | Manually queued for attention | No | +| `block_flag` | Censored / age-restricted | No | +| `split_flag` | Clip extracted from a parent video | No | +| `orig_parent_obj` | Where the video originally came from | No | +| `subs_list` | Available subtitle languages | Yes (`subtitles: Vec`) | +| `stamp_list` | Chapter timestamps (manual or extracted) | Partial — we read chapters from info.json, no manual editing | +| `slice_list` | Per-video SponsorBlock data | No — we use `--sponsorblock-mark` at download time | +| `comment_list` | Comments (yt-dlp `--write-comments`) | No | +| `error_list`, `warning_list` | Last download attempt messages | No persisted; we have rolling Job log only | + +### 2.3 `media.Channel` / `media.Playlist` / `media.Folder` + +Container fields (shared via `GenericContainer`): + +- `dbid`, `name`, `nickname`, `natname`, `parent_obj`, `child_list`. +- `options_obj` — per-container `OptionsManager` override (§3). +- Counters: `vid_count`, `bookmark_count`, `dl_count`, `fav_count`, + `live_count`, `missing_count`, `new_count`, `waiting_count`. +- `fav_flag` — cascades to all descendants. +- `error_list`, `warning_list`. +- `external_dir` — optional override storage path. +- `master_dbid` / `slave_dbid_list` — alias resolution (a video uploaded + to two channels with different IDs can be deduplicated by pointing one + to the other). + +Channels & Playlists additionally have: + +- `source` (URL). +- `rss` — RSS feed URL (for fast new-video discovery). +- `playlist_id_dict` — set of playlist IDs already imported. +- `dl_no_db_flag` — download to disk but skip the database. +- `dl_sim_flag` — always simulate (check-only, never actually fetch). +- `dl_disable_flag` — skip during operations. + +### 2.4 Storage + +- **`tartube.db`** — Python `pickle` of the entire media registry, not + SQLite. Atomic save-on-modify with periodic backups (configurable + `db_backup_mode = 'always' | 'every_session' | 'daily' | 'never'`). +- **Per-channel folder layout**: + ``` + / + / + .mp4 + .info.json + .description + ...vtt + .ytdl-archive (yt-dlp's downloaded-IDs record) + ... + ``` +- **External directories** — a channel can be redirected to a separate + drive via `external_dir`. + +**What we can do:** Pickle is a misfeature — schema migrations are +nightmarish and a corrupt pickle locks out the whole library. Our +SQLite-backed approach is strictly better. Don't adopt the storage +model; do adopt the per-channel layout (which matches ours already). + +## 3. `OptionsManager` — the headline feature + +`options.OptionsManager` is the single biggest gap between us and +Tartube. It holds **164 distinct yt-dlp / youtube-dl flags** and can +be attached to: + +- A specific `Video`. +- A `Channel` / `Playlist` / `Folder`. +- The application globally (the "default" manager). + +Resolution at download time: `Video.options_obj` → `parent.options_obj` +→ ancestor chain → app global. First non-`None` wins. A user can +create *named* OptionsManager instances and apply the same one to many +channels. + +### 3.1 Canonical option groups + +From the docstring at `options.py:69`: + +- **Behaviour**: ignore_errors, abort_on_error, live_from_start, wait_for_video_min. +- **Network**: proxy, socket_timeout, source_address, force_ipv4/6, + geo_verification_proxy, geo_bypass*, geo_bypass_country, geo_bypass_ip_block. +- **Playlist selection**: playlist_start, playlist_end, playlist_items, + max_downloads, playlist_reverse, playlist_random, + skip_playlist_after_errors, break_on_existing, break_on_reject. +- **Filters**: min_filesize, max_filesize, date, date_before, date_after, + min_views, max_views, match_filter, age_limit, include_ads, + match_title_list, reject_title_list. +- **Download**: limit_rate, retries, abort_on_unavailable_fragment, + native_hls, hls_prefer_ffmpeg, external_downloader, external_arg_string, + concurrent_fragments, throttled_rate. +- **Filesystem**: restrict_filenames, nomtime, write_description, + write_info, write_annotations, cookies_path, write_thumbnail, + force_encoding, output_format, output_template, output_format_list, + output_path_list, windows_filenames, trim_filenames, no_overwrites, + force_overwrites. +- **Auth**: username, password, two_factor, net_rc, video_password, + ap_mso, ap_username, ap_password. +- **Anti-bot**: no_check_certificate, prefer_insecure, user_agent, + referer, min_sleep_interval, max_sleep_interval. +- **Format**: video_format, all_formats, prefer_free_formats, yt_skip_dash, + merge_output_format, video_format_list, video_format_mode. +- **Subtitles**: write_subs, write_auto_subs, write_all_subs, subs_format, + subs_lang, subs_lang_list. +- **Post-processing**: extract_audio, audio_format, audio_quality, + recode_video, pp_args, keep_video, embed_subs, embed_thumbnail, + add_metadata, fixup_policy, prefer_avconv, prefer_ffmpeg. +- **Cookies**: no_cookies, cookies_from_browser, no_cookies_from_browser. +- **Output sidecars**: write_link, write_url_link, write_webloc_link, + write_desktop_link. +- **Tartube-specific**: keep_description / keep_info / keep_thumbnail / + keep_annotations (move sidecars vs delete), sim_keep_* (same in + simulate mode), move_description (move out of data dir after + download), extra_cmd_string (raw passthrough), direct_cmd_flag / + direct_url_flag (escape hatches), fetch_formats_cmd_string, + fetch_subtitles_cmd_string, downloader_config (write a yt-dlp config + file), check_fetch_comments / dl_fetch_comments / + store_comments_in_db, extractor_args_list. + +### 3.2 `OptionsParser` + +A second class that walks the canonical option list (a registry of +`OptionHolder` entries) and emits a yt-dlp CLI: `--retries 30 +--match-filter "duration > 60" -o "