freeze-watcher

A systemd daemon that continuously samples Linux system state and dumps a detailed forensic snapshot whenever freeze-like conditions are detected (D-state spikes, PSI pressure, blocked processes, load avg). Includes a Qt GUI for live metrics and snapshot browsing.

Built to diagnose intermittent system freezes that don't show up in normal resource monitors — specifically the kind where some apps stall but the mouse keeps moving.

What it captures

Each snapshot is a single text file in /var/log/freeze-watcher/snap-YYYYMMDD-HHMMSS.log containing:

loadavg, procs_running, procs_blocked, intr, ctxt
PSI cpu / io / memory pressure (some and full averages)
D-state processes with kernel wchan (where exactly they're stuck)
Top 15 by CPU and RSS
meminfo summary
iostat -xt snapshot
Top interrupt sources by total + per-CPU breakdown
Full softirq table
Last 50 dmesg lines
Last 60s of journal warnings/errors

The wchan column is the key one — that's how you tell btrfs_lock_root_node lock contention from usb_sg_wait USB stalls from folio_wait_bit_common page-fault stalls.

Components

Path	What
`src/freeze-watcher.sh`	The daemon — samples every 1s, captures on threshold
`src/freeze-monitor-gui`	PySide6 GUI: live metrics + snapshot browser + tray icon
`systemd/freeze-watcher.service`	systemd unit
`desktop/freeze-monitor.desktop`	App launcher entry

Install

Via Makefile (system-wide install to `/usr/local`)

sudo make install
sudo make enable

As an Arch package

./packaging/build-package.sh
sudo pacman -U freeze-watcher-*.pkg.tar.zst
sudo systemctl enable --now freeze-watcher.service

Update workflow

When you change something:

# 1. Edit files in src/ or systemd/
# 2. Reinstall
sudo make install

# 3. If the daemon script changed, restart it
sudo systemctl restart freeze-watcher.service

# 4. If the GUI changed, just relaunch it (it's a regular Qt app)

If you're using the Arch package: ./packaging/build-package.sh && sudo pacman -U *.pkg.tar.zst will rebuild and upgrade in place.

Tuning thresholds

Open src/freeze-watcher.sh and edit the constants at the top:

PSI_IO_THRESHOLD=40       # io pressure avg10 % to trigger
PSI_CPU_THRESHOLD=50      # cpu pressure avg10 % to trigger
DSTATE_THRESHOLD=5        # processes in D state
BLOCKED_THRESHOLD=5       # procs_blocked from /proc/stat
LOAD_THRESHOLD=20         # 1-min load average
COOLDOWN=30               # seconds between captures
SAMPLE_INTERVAL=1         # seconds between samples

GUI

Launch from app menu (search "Freeze Watcher") or freeze-monitor-gui.

Live PSI / D-state / load metrics with green/orange/red color coding (1s refresh)
Service status with start/stop buttons (uses pkexec)
Captured event list — click any row for full snapshot view
Closes to system tray; quit from the tray menu
Notification when a new snapshot is captured

Reading a snapshot

The most informative section is D-state processes — the wchan tells you what kernel function each process is blocked in. Common ones:

wchan	Meaning
`btrfs_lock_root_node`	btrfs metadata lock contention (often qgroups + snapshots)
`btrfs_search_slot`	btrfs tree walk waiting for lock
`btrfs_sync_log`	fsync waiting for transaction commit
`folio_wait_bit_common`	Waiting for memory page (often swap-in)
`usb_sg_wait`	USB device stall
`io_schedule`	Generic I/O wait

Cross-reference wchan with PSI io and PSI cpu to distinguish lock contention (low PSI) from genuine resource pressure (high PSI).

Future ideas

Per-cgroup PSI capture
eBPF tracing for long-running syscalls
Web UI option (no Qt requirement)
Auto-prune old snapshots
Configurable /etc/freeze-watcher.conf instead of inline thresholds
Integration with desktop notifications via notify-send
Optional perf-profile capture during event

License

MIT

3.9 KiB Raw Blame History