Perceptual-hash dedup: fingerprint engine + storage (3.7, part 1)
The core of content-aware duplicate detection — finding the same video under a different ID (reupload / mirror / re-encode), which the ID-based maintenance scan can't. - src/fingerprint.rs: samples 6 frames per video via ffmpeg keyframe seek (fast — never full-decodes), downscales to a 9x8 grayscale grid, and dHashes each to a 64-bit frame hash. `group_similar` clusters videos whose frame hashes match within a Hamming threshold, comparing only within a duration-tolerance window (sorted sliding window + union-find) so it stays near-linear instead of O(n²). Parallel `compute_batch` over a hand-rolled scoped thread pool (no rayon), with a progress counter. - src/database.rs: `video_fingerprint` table keyed by (path, mtime) like info_cache — hash once, reuse forever; new downloads hashed as they arrive. Store / load / mtime-gate / prune helpers. Unit tests cover dHash, Hamming, duration-bucketed + transitive grouping, and DB round-trip/replace/prune. An opt-in `real_ffmpeg_groups_reencodes` test (run with --ignored) generates a video + a downscaled CRF-38 re-encode + an unrelated clip and asserts the first two group and the third doesn't — measured ~0.5s/video serial (≈65ms/video across 8 cores). UI wiring (background build + review screen in both front-ends) follows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
fcd202fde0
commit
6d5c1cae33
3 changed files with 489 additions and 0 deletions
|
|
@ -21,6 +21,7 @@ mod disk_space;
|
|||
mod download_options;
|
||||
mod downloader;
|
||||
mod error_class;
|
||||
mod fingerprint;
|
||||
mod library;
|
||||
mod maintenance;
|
||||
mod platform;
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue