Perceptual-hash dedup: fingerprint engine + storage (3.7, part 1)

The core of content-aware duplicate detection — finding the same video
under a different ID (reupload / mirror / re-encode), which the ID-based
maintenance scan can't.

- src/fingerprint.rs: samples 6 frames per video via ffmpeg keyframe seek
  (fast — never full-decodes), downscales to a 9x8 grayscale grid, and
  dHashes each to a 64-bit frame hash. `group_similar` clusters videos
  whose frame hashes match within a Hamming threshold, comparing only
  within a duration-tolerance window (sorted sliding window + union-find)
  so it stays near-linear instead of O(n²). Parallel `compute_batch` over
  a hand-rolled scoped thread pool (no rayon), with a progress counter.
- src/database.rs: `video_fingerprint` table keyed by (path, mtime) like
  info_cache — hash once, reuse forever; new downloads hashed as they
  arrive. Store / load / mtime-gate / prune helpers.

Unit tests cover dHash, Hamming, duration-bucketed + transitive grouping,
and DB round-trip/replace/prune. An opt-in `real_ffmpeg_groups_reencodes`
test (run with --ignored) generates a video + a downscaled CRF-38
re-encode + an unrelated clip and asserts the first two group and the
third doesn't — measured ~0.5s/video serial (≈65ms/video across 8 cores).

UI wiring (background build + review screen in both front-ends) follows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Luna 2026-06-07 06:03:53 -07:00
parent fcd202fde0
commit 6d5c1cae33
3 changed files with 489 additions and 0 deletions

View file

@ -21,6 +21,7 @@ mod disk_space;
mod download_options;
mod downloader;
mod error_class;
mod fingerprint;
mod library;
mod maintenance;
mod platform;