freeze-watcher

A systemd daemon that continuously samples Linux system state and dumps a detailed forensic snapshot whenever freeze-like conditions are detected (D-state spikes, PSI pressure, blocked processes, load avg). Includes a Qt GUI for live metrics and snapshot browsing.

Built to diagnose intermittent system freezes that don't show up in normal resource monitors — specifically the kind where some apps stall but the mouse keeps moving.

What it captures

Each snapshot is a single text file in /var/log/freeze-watcher/snap-YYYYMMDD-HHMMSS.log containing:

• loadavg, procs_running, procs_blocked, intr, ctxt
• PSI cpu / io / memory pressure (some and full averages)
• D-state processes with kernel wchan (where exactly they're stuck)
• Top 15 by CPU and RSS
• meminfo summary
• iostat -xt snapshot
• Top interrupt sources by total + per-CPU breakdown
• Full softirq table
• Last 50 dmesg lines
• Last 60s of journal warnings/errors

The wchan column is the key one — that's how you tell btrfs_lock_root_node lock contention from usb_sg_wait USB stalls from folio_wait_bit_common page-fault stalls.

Components

Path	What
src/freeze-watcher.sh	The daemon — samples every 1s, captures on threshold
src/freeze-monitor-gui	PySide6 GUI: live metrics + snapshot browser + tray icon
systemd/freeze-watcher.service	systemd unit
desktop/freeze-monitor.desktop	App launcher entry

Install

Via Makefile (system-wide install to /usr/local)

sudo make install
sudo make enable

As an Arch package

./packaging/build-package.sh
sudo pacman -U freeze-watcher-*.pkg.tar.zst
sudo systemctl enable --now freeze-watcher.service

Update workflow

When you change something:

# 1. Edit files in src/ or systemd/
# 2. Reinstall
sudo make install

# 3. If the daemon script changed, restart it
sudo systemctl restart freeze-watcher.service

# 4. If the GUI changed, just relaunch it (it's a regular Qt app)

If you're using the Arch package: ./packaging/build-package.sh && sudo pacman -U *.pkg.tar.zst will rebuild and upgrade in place.

Configuration

Edit /etc/freeze-watcher.conf. After changes:

sudo systemctl restart freeze-watcher.service

Options:

Variable	Default	Purpose
PSI_IO_THRESHOLD	40	io pressure avg10 % to trigger
PSI_CPU_THRESHOLD	50	cpu pressure avg10 % to trigger
DSTATE_THRESHOLD	5	processes in uninterruptible sleep
BLOCKED_THRESHOLD	5	procs_blocked from /proc/stat
LOAD_THRESHOLD	20	1-min load average
SAMPLE_INTERVAL	1	seconds between samples
COOLDOWN	30	min seconds between captures
MAX_SNAPSHOTS	200	auto-delete oldest beyond this count (0 disables)
MAX_SNAPSHOT_AGE_DAYS	30	auto-delete older than this (0 disables)
NOTIFY_USERS	""	comma-separated users to notify-send on capture
NOTIFY_URGENCY	normal	low / normal / critical
CAPTURE_KERNEL_STACKS	1	dump /proc/[pid]/stack for D-state procs
CAPTURE_BPFTRACE	0	run bpftrace one-shot (requires bpftrace)
CAPTURE_CGROUP_PSI	1	walk cgroups for top per-cgroup PSI

GUI

Launch from app menu (search "Freeze Watcher") or freeze-monitor-gui.

• Live PSI / D-state / load metrics with green/orange/red color coding (1s refresh)
• Service status with start/stop buttons (uses pkexec)
• Captured event list — click any row for full snapshot view
• Closes to system tray; quit from the tray menu
• Notification when a new snapshot is captured

Reading a snapshot

The most informative section is D-state processes — the wchan tells you what kernel function each process is blocked in. Common ones:

wchan	Meaning
btrfs_lock_root_node	btrfs metadata lock contention (often qgroups + snapshots)
btrfs_search_slot	btrfs tree walk waiting for lock
btrfs_sync_log	fsync waiting for transaction commit
folio_wait_bit_common	Waiting for memory page (often swap-in)
usb_sg_wait	USB device stall
io_schedule	Generic I/O wait

Cross-reference wchan with PSI io and PSI cpu to distinguish lock contention (low PSI) from genuine resource pressure (high PSI).

Auto-classification

Each captured snapshot is automatically pattern-matched against known freeze signatures and gets a ## Diagnosis section at the top:

• btrfs lock contention — btrfs_lock_root_node / btrfs_sync_log wchans with low PSI io
• btrfs metadata pressure — btrfs_search_slot without lock contention
• USB storage stall — usb_sg_wait, scsi_eh, sd_check_events
• Memory swap thrash — folio_wait_bit_common with PSI mem
• Page-fault stall — folio_wait_bit_common without mem pressure (overly aggressive swappiness)
• CPU oversubscription — high PSI cpu, few D-state procs
• NIC IRQ saturation — recent NET_RX softirq concentrated on one CPU (computed against a rolling 5-minute baseline)
• Genuine I/O bottleneck — high PSI io.full

If no signature matches, the snapshot says "uncertain — review manually".

Cross-snapshot summary

The GUI's Summary tab aggregates patterns across all captured snapshots:

• Top diagnoses (which freeze types occur most)
• Top kernel wchans (where processes get stuck)
• Top processes seen in D-state
• Top CPU consumers at capture time
• Captures by hour-of-day distribution

Useful for spotting "qBittorrent appears in 80% of freezes" type patterns.

Live graphs

The GUI's Live graphs tab shows the last hour of load, PSI io/cpu, and D-state count as scrolling line graphs. Red dashed lines mark snapshot capture events so you can correlate spikes with the captured forensics.

Manual capture

Click "📸 Capture Now" in the GUI status bar (or in the tray menu) to trigger a snapshot immediately, regardless of whether thresholds are crossed. Useful when you suspect something is brewing but it hasn't tripped the alarms.

Future ideas

• Web UI option (no Qt requirement)
• Optional perf-profile capture during event
• HTML/JSON snapshot output mode
• Snapshot annotations (user notes per capture)
• Anonymized export for sharing

License

MIT