163 lines
6.1 KiB
Markdown
163 lines
6.1 KiB
Markdown
# freeze-watcher
|
|
|
|
A systemd daemon that continuously samples Linux system state and dumps a detailed forensic snapshot whenever freeze-like conditions are detected (D-state spikes, PSI pressure, blocked processes, load avg). Includes a Qt GUI for live metrics and snapshot browsing.
|
|
|
|
Built to diagnose intermittent system freezes that don't show up in normal resource monitors — specifically the kind where some apps stall but the mouse keeps moving.
|
|
|
|
## What it captures
|
|
|
|
Each snapshot is a single text file in `/var/log/freeze-watcher/snap-YYYYMMDD-HHMMSS.log` containing:
|
|
|
|
- `loadavg`, `procs_running`, `procs_blocked`, `intr`, `ctxt`
|
|
- PSI cpu / io / memory pressure (`some` and `full` averages)
|
|
- D-state processes with kernel `wchan` (where exactly they're stuck)
|
|
- Top 15 by CPU and RSS
|
|
- `meminfo` summary
|
|
- `iostat -xt` snapshot
|
|
- Top interrupt sources by total + per-CPU breakdown
|
|
- Full `softirq` table
|
|
- Last 50 dmesg lines
|
|
- Last 60s of journal warnings/errors
|
|
|
|
The wchan column is the key one — that's how you tell `btrfs_lock_root_node` lock contention from `usb_sg_wait` USB stalls from `folio_wait_bit_common` page-fault stalls.
|
|
|
|
## Components
|
|
|
|
| Path | What |
|
|
|---|---|
|
|
| `src/freeze-watcher.sh` | The daemon — samples every 1s, captures on threshold |
|
|
| `src/freeze-monitor-gui` | PySide6 GUI: live metrics + snapshot browser + tray icon |
|
|
| `systemd/freeze-watcher.service` | systemd unit |
|
|
| `desktop/freeze-monitor.desktop` | App launcher entry |
|
|
|
|
## Install
|
|
|
|
### Via Makefile (system-wide install to `/usr/local`)
|
|
|
|
```
|
|
sudo make install
|
|
sudo make enable
|
|
```
|
|
|
|
### As an Arch package
|
|
|
|
```
|
|
./packaging/build-package.sh
|
|
sudo pacman -U freeze-watcher-*.pkg.tar.zst
|
|
sudo systemctl enable --now freeze-watcher.service
|
|
```
|
|
|
|
## Update workflow
|
|
|
|
When you change something:
|
|
|
|
```
|
|
# 1. Edit files in src/ or systemd/
|
|
# 2. Reinstall
|
|
sudo make install
|
|
|
|
# 3. If the daemon script changed, restart it
|
|
sudo systemctl restart freeze-watcher.service
|
|
|
|
# 4. If the GUI changed, just relaunch it (it's a regular Qt app)
|
|
```
|
|
|
|
If you're using the Arch package: `./packaging/build-package.sh && sudo pacman -U *.pkg.tar.zst` will rebuild and upgrade in place.
|
|
|
|
## Configuration
|
|
|
|
Edit `/etc/freeze-watcher.conf`. After changes:
|
|
|
|
```
|
|
sudo systemctl restart freeze-watcher.service
|
|
```
|
|
|
|
Options:
|
|
|
|
| Variable | Default | Purpose |
|
|
|---|---|---|
|
|
| `PSI_IO_THRESHOLD` | 40 | io pressure avg10 % to trigger |
|
|
| `PSI_CPU_THRESHOLD` | 50 | cpu pressure avg10 % to trigger |
|
|
| `DSTATE_THRESHOLD` | 5 | processes in uninterruptible sleep |
|
|
| `BLOCKED_THRESHOLD` | 5 | procs_blocked from /proc/stat |
|
|
| `LOAD_THRESHOLD` | 20 | 1-min load average |
|
|
| `SAMPLE_INTERVAL` | 1 | seconds between samples |
|
|
| `COOLDOWN` | 30 | min seconds between captures |
|
|
| `MAX_SNAPSHOTS` | 200 | auto-delete oldest beyond this count (0 disables) |
|
|
| `MAX_SNAPSHOT_AGE_DAYS` | 30 | auto-delete older than this (0 disables) |
|
|
| `NOTIFY_USERS` | "" | comma-separated users to notify-send on capture |
|
|
| `NOTIFY_URGENCY` | normal | low / normal / critical |
|
|
| `CAPTURE_KERNEL_STACKS` | 1 | dump /proc/[pid]/stack for D-state procs |
|
|
| `CAPTURE_BPFTRACE` | 0 | run bpftrace one-shot (requires bpftrace) |
|
|
| `CAPTURE_CGROUP_PSI` | 1 | walk cgroups for top per-cgroup PSI |
|
|
|
|
## GUI
|
|
|
|
Launch from app menu (search "Freeze Watcher") or `freeze-monitor-gui`.
|
|
|
|
- Live PSI / D-state / load metrics with green/orange/red color coding (1s refresh)
|
|
- Service status with start/stop buttons (uses `pkexec`)
|
|
- Captured event list — click any row for full snapshot view
|
|
- Closes to system tray; quit from the tray menu
|
|
- Notification when a new snapshot is captured
|
|
|
|
## Reading a snapshot
|
|
|
|
The most informative section is **D-state processes** — the `wchan` tells you what kernel function each process is blocked in. Common ones:
|
|
|
|
| wchan | Meaning |
|
|
|---|---|
|
|
| `btrfs_lock_root_node` | btrfs metadata lock contention (often qgroups + snapshots) |
|
|
| `btrfs_search_slot` | btrfs tree walk waiting for lock |
|
|
| `btrfs_sync_log` | fsync waiting for transaction commit |
|
|
| `folio_wait_bit_common` | Waiting for memory page (often swap-in) |
|
|
| `usb_sg_wait` | USB device stall |
|
|
| `io_schedule` | Generic I/O wait |
|
|
|
|
Cross-reference `wchan` with PSI io and PSI cpu to distinguish lock contention (low PSI) from genuine resource pressure (high PSI).
|
|
|
|
## Auto-classification
|
|
|
|
Each captured snapshot is automatically pattern-matched against known freeze signatures and gets a `## Diagnosis` section at the top:
|
|
|
|
- **btrfs lock contention** — `btrfs_lock_root_node` / `btrfs_sync_log` wchans with low PSI io
|
|
- **btrfs metadata pressure** — `btrfs_search_slot` without lock contention
|
|
- **USB storage stall** — `usb_sg_wait`, `scsi_eh`, `sd_check_events`
|
|
- **Memory swap thrash** — `folio_wait_bit_common` with PSI mem
|
|
- **Page-fault stall** — `folio_wait_bit_common` without mem pressure (overly aggressive swappiness)
|
|
- **CPU oversubscription** — high PSI cpu, few D-state procs
|
|
- **NIC IRQ saturation** — recent NET_RX softirq concentrated on one CPU (computed against a rolling 5-minute baseline)
|
|
- **Genuine I/O bottleneck** — high PSI io.full
|
|
|
|
If no signature matches, the snapshot says "uncertain — review manually".
|
|
|
|
## Cross-snapshot summary
|
|
|
|
The GUI's **Summary** tab aggregates patterns across all captured snapshots:
|
|
- Top diagnoses (which freeze types occur most)
|
|
- Top kernel wchans (where processes get stuck)
|
|
- Top processes seen in D-state
|
|
- Top CPU consumers at capture time
|
|
- Captures by hour-of-day distribution
|
|
|
|
Useful for spotting "qBittorrent appears in 80% of freezes" type patterns.
|
|
|
|
## Live graphs
|
|
|
|
The GUI's **Live graphs** tab shows the last hour of load, PSI io/cpu, and D-state count as scrolling line graphs. Red dashed lines mark snapshot capture events so you can correlate spikes with the captured forensics.
|
|
|
|
## Manual capture
|
|
|
|
Click "📸 Capture Now" in the GUI status bar (or in the tray menu) to trigger a snapshot immediately, regardless of whether thresholds are crossed. Useful when you suspect something is brewing but it hasn't tripped the alarms.
|
|
|
|
## Future ideas
|
|
|
|
- Web UI option (no Qt requirement)
|
|
- Optional perf-profile capture during event
|
|
- HTML/JSON snapshot output mode
|
|
- Snapshot annotations (user notes per capture)
|
|
- Anonymized export for sharing
|
|
|
|
## License
|
|
|
|
MIT
|