# Enodia Sentinel Incident Response Runbooks Practical, conservative response playbooks for the alerts Sentinel raises. Each runbook follows the same arc: **confirm → preserve → contain → recover**, and prefers reversible, manual steps. Sentinel does not perform automatic containment (see [THREAT_MODEL.md](THREAT_MODEL.md)); these are operator actions. > A note on trust: if you suspect kernel-level compromise, remember that the > tools you run (`ss`, `ps`, even `ls`) may be lying to you. Cross-check from the > dashboard on another machine, and when in doubt, capture evidence and analyze > the disk offline. ## The First Five Minutes For any alert, before changing anything: 1. **Read the snapshot.** Every alert writes `alert-*.log` (human) and `alert-*.json` (machine) under `/var/log/enodia-sentinel`. Note the `signature`, `sid`, severity, PIDs, executable paths, and remote peers. 2. **Get host state.** `enodia-sentinel status` — is the daemon healthy, how many alerts, when was the last. 3. **Triage known noise.** `enodia-sentinel triage` separates likely false positives from findings that need a human. 4. **Preserve evidence before you touch anything:** ```bash tar -C /var/log -czf /tmp/enodia-evidence-$(date +%s).tgz enodia-sentinel ``` Then pick the runbook below that matches the signature. --- ## Runbook 1 — Reverse shell / suspicious egress **Triggers:** `reverse_shell` (sid 100010), `egress` (sid 100016). An interpreter wired to a socket, or an interpreter holding a connection to a public IP — the classic interactive-C2 and beaconing shapes. **Confirm** - Open the snapshot; identify the PID, `comm`, parent process, and remote peer. - Inspect live (these views may be untrustworthy if root is compromised): ```bash ps -o pid,ppid,user,comm,args -p ls -l /proc//fd # which fds are the socket cat /proc//environ | tr '\0' '\n' ss -tnp | grep # the peer ``` - Is the peer expected? Compare against `egress_allow_cidrs` in config. A package manager or browser to a public IP is normal; `bash`/`python`/`nc` is not. **Contain** - Capture the process before killing it (memory/maps if you have tooling), then: `kill -STOP ` to freeze, inspect, and only then `kill -9`. - Block the peer if it is hostile: `nft add rule inet filter output ip daddr drop`. - Find persistence the shell may have dropped — run Runbook 2. **Recover** - Rotate any credentials that were reachable from the compromised process. - Rebuild listener/SUID baselines only after the host is confirmed clean. **False positives:** legitimate admin scripts, monitoring agents, or P2P clients. Add trusted egress destinations to `egress_allow_cidrs` rather than disabling the detector. --- ## Runbook 2 — Persistence tampering **Triggers:** `persistence` (sid 100015), `fim_modified` on a watched persistence path. A change to cron, a systemd unit, an SSH `authorized_keys`, a shell rc file, or sudo/account config — how an attacker survives a reboot. **Confirm** - The snapshot names the changed path. Diff it against the FIM baseline and package ownership: ```bash enodia-sentinel fim-check enodia-sentinel fim-check --packages # does a package own & vouch for it? ``` - For SSH keys: is the added key yours? For systemd/cron: does the unit run a binary in a writable or unusual location? **Contain** - Remove the unauthorized entry (the added key, cron line, or unit). Keep a copy in your evidence bundle first. - `systemctl disable --now ` for a malicious service; check what it spawned. **Recover** - After removing persistence, hunt for the process it launched (Runbook 1) and any trojaned binaries (Runbook 3). - Re-anchor the baseline only once clean: `enodia-sentinel fim-update`. **False positives:** legitimate package upgrades and admin edits. The pacman hook auto-runs `fim-update` for normal transactions; an unexplained change outside a transaction is the suspicious case. --- ## Runbook 3 — Trojaned binary / package tampering **Triggers:** `fim_modified` / `fim_pkg_modified` on a system binary, `pkgdb_tamper` (package DB edited out of band), `pkg_signature_mismatch` (on-disk file ≠ signed package), `pacman_siglevel_disabled`. **Confirm** - Layer the integrity checks from weakest to strongest trust anchor: ```bash enodia-sentinel fim-check --packages # vs FIM baseline + pacman -Qkk enodia-sentinel pkgdb-check # was the local checksum DB rewritten? enodia-sentinel pkgdb-verify --sample 200 # vs signed cache packages (.MTREE) ``` - `pkg_signature_mismatch` is the strongest signal: it bypasses the mutable local DB. Treat a mismatch on a system binary as a trojaned file until proven otherwise. `pkgdb_tamper` suggests someone tried to hide exactly this. **Contain** - Do not execute the suspect binary. If it is currently running, freeze it (Runbook 1). - Confirm package-signature policy is intact: `enodia-sentinel posture check` (flags `pacman_siglevel_disabled`). **Recover** - Identify the owning package and reinstall from a trusted source: ```bash pacman -Qo /path/to/binary pacman -S --overwrite '*' # reinstall the verified file enodia-sentinel fim-check --packages # confirm it now matches ``` - Then re-anchor: `enodia-sentinel fim-update` and re-run `pkgdb-check`. **False positives:** a mid-upgrade snapshot can momentarily disagree. Re-run after the transaction completes; a persistent mismatch is the real finding. --- ## Runbook 4 — Hidden listener / rootkit **Triggers:** `rootkit_hidden_process` (100022), `rootkit_hidden_module` (100023), `rootkit_hidden_port` (100024), `promiscuous_interface` (100025), and `new_listener` (100013) for an unexplained open port. These come from cross-view checks: the same question asked two ways, answered differently, because something is hiding. **Confirm** - Re-run the cross-view sweep and capture it: ```bash enodia-sentinel rootcheck ``` - `new_listener`: identify the owning process and whether its binary is package-owned. A hidden port that `rootcheck` sees but `ss` does not is far more serious than a forgotten service. - A `rootkit_hidden_module` names a live kernel module absent from `/proc/modules` — strong evidence of an LKM rootkit. **Contain** - Assume user-space tools are compromised. Pivot to the read-only dashboard from **another machine** to cross-check, and prefer offline analysis. - Do not `rmmod` blindly — capture the module and system state first. **Recover** - A confirmed kernel rootkit means **rebuild from known-good media**; you cannot reliably clean a host whose kernel is hooked. Preserve the disk image first. **False positives:** PID exit races (Sentinel re-verifies hidden PIDs to reduce these) and a legitimately promiscuous interface (a sniffer/monitoring tool you run). Confirm the owning process before escalating. --- ## Runbook 5 — Sensor tampering **Triggers:** `fim_modified` on a Sentinel-owned path (self-integrity), a stale or missing heartbeat surfaced by the off-box `watchdog`, or the dashboard going unreachable. Disabling the alarm is step zero of a competent intrusion — so a silent or edited Sentinel is itself an alert. **Confirm** - From the **watchdog host** (never the protected host): ```bash enodia-sentinel watchdog --url http://:8787 --token --max-age 120 ``` Exit 1 means down, unreachable, or stale. - On the host (if you still trust it): `enodia-sentinel status` and check whether Sentinel's own binaries/config/units/hook changed: ```bash enodia-sentinel fim-check # SELF_PATHS are always watched ``` **Contain** - If Sentinel's files were edited, treat the host as compromised: an attacker with the privilege to edit the sensor had the privilege to do far more. - Move to evidence-preservation and offline analysis rather than trusting any further local output. **Recover** - Reinstall Sentinel from a trusted source and re-baseline on a clean host. - Harden against recurrence: `chattr +i` on Sentinel's binaries, config, units, baselines, and the pacman hook; keep the watchdog running off-box. **False positives:** a legitimate Sentinel upgrade changes its own files — but that should coincide with a package transaction. An edit outside one is the suspicious case. --- ## After any incident - Keep the evidence bundle; do not delete snapshots while an investigation is open (retention is bounded by `max_snapshots` / `max_snapshot_age_days`). - Re-baseline (`baseline`, `fim-update`) only once the host is confirmed clean — baselining a compromised host blesses the compromise. - Re-run `enodia-sentinel posture check` to close the hygiene gaps that enabled the intrusion. - The incident-grouping commands (`incident show/export`) on the [roadmap](ROADMAP.md) will eventually automate the evidence-bundle steps above.