# Enodia Sentinel Incident Response Runbooks Practical, conservative response playbooks for the alerts Sentinel raises. Each runbook follows the same arc: **confirm → preserve → contain → recover**, and prefers reversible, manual steps. Sentinel does not perform automatic containment (see [THREAT_MODEL.md](THREAT_MODEL.md)); these are operator actions. > A note on trust: if you suspect kernel-level compromise, remember that the > tools you run (`ss`, `ps`, even `ls`) may be lying to you. Cross-check from the > dashboard on another machine, and when in doubt, capture evidence and analyze > the disk offline. ## The First Five Minutes For any alert, before changing anything: 1. **Read the snapshot.** Every alert writes `alert-*.log` (human) and `alert-*.json` (machine) under `/var/log/enodia-sentinel`. Note the `signature`, `sid`, severity, PIDs, executable paths, and remote peers. 2. **Get host state.** `enodia-sentinel status` — is the daemon healthy, how many alerts, when was the last. 3. **Triage known noise.** `enodia-sentinel triage` separates likely false positives from findings that need a human. 4. **Build a dry-run response plan.** Find the incident and review the proposed actions: ```bash enodia-sentinel incident list enodia-sentinel respond plan ``` The CLI writes a copy under `/var/log/enodia-sentinel/response-plans/` and appends `/var/log/enodia-sentinel/response-audit.log`, so the reviewed plan can be handed off or compared with later actions. 5. **Preserve evidence before you touch anything:** ```bash tar -C /var/log -czf /tmp/enodia-evidence-$(date +%s).tgz enodia-sentinel ``` Then pick the runbook below that matches the signature. --- ## Runbook 1 — Reverse shell / suspicious egress **Triggers:** `reverse_shell` (sid 100010), `egress` (sid 100016), `stealth_network` (sid 100036). An interpreter wired to a socket, an interpreter holding a connection to a public IP, or activity over raw/SCTP/DCCP/packet/XDP/TIPC/vsock families - the classic interactive-C2, beaconing, and covert-channel shapes. **Confirm** - Open the snapshot; identify the PID, `comm`, parent process, and remote peer. - Inspect live (these views may be untrustworthy if root is compromised): ```bash ps -o pid,ppid,user,comm,args -p ls -l /proc//fd # which fds are the socket cat /proc//environ | tr '\0' '\n' ss -tnp | grep # the peer ``` - Is the peer expected? Compare against `egress_allow_cidrs` in config. A package manager or browser to a public IP is normal; `bash`/`python`/`nc` is not. - For `stealth_network`, check the socket family and owner. Packet/raw sockets are normal for approved sniffers and some network managers; SCTP/DCCP/TIPC/XDP on a desktop or ordinary server should have a clear service owner. Tune with `stealth_network_allow_comms` or, for intentionally used families, `stealth_network_allow_kinds`. **Contain** - Capture the process before killing it (memory/maps if you have tooling), then: `kill -STOP ` to freeze, inspect, and only then `kill -9`. - Block the peer if it is hostile: `nft add rule inet filter output ip daddr drop`. - Find persistence the shell may have dropped — run Runbook 2. **Recover** - Rotate any credentials that were reachable from the compromised process. - Rebuild listener/SUID baselines only after the host is confirmed clean. **False positives:** legitimate admin scripts, monitoring agents, or P2P clients. Add trusted egress destinations to `egress_allow_cidrs` rather than disabling the detector. --- ## Runbook 2 — Persistence tampering **Triggers:** `persistence` (sid 100015), `fim_modified` on a watched persistence path, `systemd_unit_writable` (sid 100050), `systemd_exec_unsafe_path` (sid 100051). A change to cron, a systemd unit, an SSH `authorized_keys`, a shell rc file, or sudo/account config — how an attacker survives a reboot. **Confirm** - The snapshot names the changed path. Diff it against the FIM baseline and package ownership: ```bash enodia-sentinel fim-check enodia-sentinel fim-check --packages # does a package own & vouch for it? ``` - For SSH keys: is the added key yours? For systemd/cron: does the unit run a binary in a writable or unusual location? **Contain** - Remove the unauthorized entry (the added key, cron line, or unit). Keep a copy in your evidence bundle first. - `systemctl disable --now ` for a malicious service; check what it spawned. **Recover** - After removing persistence, hunt for the process it launched (Runbook 1) and any trojaned binaries (Runbook 3). - Re-anchor the baseline only once clean: `enodia-sentinel fim-update`. **False positives:** legitimate package upgrades and admin edits. The pacman hook auto-runs `fim-update` for normal transactions; an unexplained change outside a transaction is the suspicious case. --- ## Runbook 3 — Trojaned binary / package tampering **Triggers:** `fim_modified` / `fim_pkg_modified` on a system binary, `pkgdb_tamper` (package DB edited out of band), `pkg_signature_mismatch` (on-disk file ≠ signed package), `pacman_siglevel_disabled`. **Confirm** - Layer the integrity checks from weakest to strongest trust anchor: ```bash enodia-sentinel fim-check --packages # vs FIM baseline + pacman -Qkk enodia-sentinel pkgdb-check # was the local checksum DB rewritten? enodia-sentinel pkgdb-verify --sample 200 # vs signed cache packages (.MTREE) ``` - `pkg_signature_mismatch` is the strongest signal: it bypasses the mutable local DB. Treat a mismatch on a system binary as a trojaned file until proven otherwise. `pkgdb_tamper` suggests someone tried to hide exactly this. **Contain** - Do not execute the suspect binary. If it is currently running, freeze it (Runbook 1). - Confirm package-signature policy is intact: `enodia-sentinel posture check` (flags `pacman_siglevel_disabled`). **Recover** - Identify the owning package and reinstall from a trusted source: ```bash enodia-sentinel respond plan pacman -Qo /path/to/binary enodia-sentinel pkgdb-verify pacman -S --overwrite '*' # reinstall the verified file enodia-sentinel fim-check --packages # confirm it now matches enodia-sentinel fim-update # re-anchor only after review ``` - Then re-run `pkgdb-check`. **False positives:** a mid-upgrade snapshot can momentarily disagree. Re-run after the transaction completes; a persistent mismatch is the real finding. --- ## Runbook 4 — Hidden listener / rootkit **Triggers:** `rootkit_hidden_process` (100022), `rootkit_hidden_module` (100023), `rootkit_hidden_port` (100024), `promiscuous_interface` (100025), `rootkit_known_module` (100028), `rootkit_tainted_module` (100029), `kernel_tainted` (100030), `rootkit_hidden_udp_port` (100031), `rootkit_hidden_raw_socket` (100034), `raw_icmp_socket` (100035), `rootkit_hidden_protocol_socket` (100037), `rootkit_ps_hidden_process` (100038), and `new_listener` (100013) for an unexplained open port. These come from cross-view checks: the same question asked two ways, answered differently, because something is hiding. **Confirm** - Re-run the cross-view sweep and capture it: ```bash enodia-sentinel rootcheck ``` - `new_listener`: identify the owning process and whether its binary is package-owned. A hidden port that `rootcheck` sees but `ss` does not is far more serious than a forgotten service. - `rootkit_ps_hidden_process`: compare `/proc//cmdline`, `/proc//exe`, and `/proc//maps` against `ps`/`pgrep` output. A process visible in `/proc` but missing from `ps` strongly suggests userland process-tool hiding. - `rootkit_hidden_udp_port`: check whether a legitimate UDP service owns the port; a port present in `/proc/net/udp*` but absent from `ss -u` suggests tool output may be hooked. - `rootkit_hidden_raw_socket` or `raw_icmp_socket`: look for authorized ping, monitoring, traceroute, or packet tooling first. A persistent raw ICMP socket with no operator explanation is consistent with ICMP knockers/backdoors and should be investigated with process fd tables, package provenance, and memory capture before killing the process. - `rootkit_hidden_protocol_socket`: inspect SCTP, DCCP, packet, TIPC, or XDP use. If `/proc/net` sees the family but `ss` does not, treat normal socket tooling as suspect and pivot to offline evidence or the dashboard from another host. - A `rootkit_hidden_module` names a live kernel module absent from `/proc/modules` — strong evidence of an LKM rootkit. - A `rootkit_known_module` finding names a module associated with public LKM rootkits; treat it as a high-confidence compromise until proven otherwise. - A `rootkit_tainted_module` or `kernel_tainted` finding may be legitimate vendor/DKMS software, but unsigned/out-of-tree or force-loaded modules should be explained and allowlisted only after review. **Contain** - Assume user-space tools are compromised. Pivot to the read-only dashboard from **another machine** to cross-check, and prefer offline analysis. - Do not `rmmod` blindly — capture the module and system state first. **Recover** - A confirmed kernel rootkit means **rebuild from known-good media**; you cannot reliably clean a host whose kernel is hooked. Preserve the disk image first. **False positives:** PID exit races (Sentinel re-verifies hidden PIDs to reduce these), a legitimately promiscuous interface (a sniffer/monitoring tool you run), and expected tainted vendor/DKMS modules. Confirm the owning process or module provenance before escalating; use `rootcheck_module_allow` only for modules you intentionally trust. --- ## Runbook 5 — Memory obfuscation, process hiding, or injection library **Triggers:** `memory_obfuscation` (100039), `process_hiding_library` (100048), `process_injection_library` (100049) These alerts come from `/proc//maps`. They do not read process memory; they flag map shapes associated with encrypted/packed payloads or userland hiding: RWX pages, executable anonymous memory, executable `memfd`/deleted mappings, or libraries with process-hiding names such as `libhide`. They also flag executable shared-library mappings from writable/runtime paths such as `/tmp`, `/dev/shm`, `/var/tmp`, and `/run/user`, which are common places for injected userland hooks or staged `.so` payloads. **Confirm** - Open the snapshot and inspect `cmdline`, `exe`, parent process, and captured memory maps for the PID. - For `process_hiding_library`, check whether the mapped library is package-owned and whether it is loaded into tools like `ps`, `pgrep`, `ss`, shells, or admin utilities. - For `process_injection_library`, preserve the mapped `.so`, check its owner, hash, parent process, and whether it was deliberately loaded by a known profiler, debugger, or local instrumentation tool. - For `memory_obfuscation`, distinguish expected JIT runtimes from unknown implants. Tune known runtimes with `memory_obfuscation_allow_comms`; use `memory_obfuscation_allow_paths` only for well-understood local runtimes or instrumentation. **Contain** - Preserve the executable and `/proc//maps` before killing. If the alert is for executable `memfd` or deleted mappings, capture memory first if you have trusted tooling; the on-disk payload may be gone. - Re-run `rootcheck` after containment to confirm no process-tool hiding or hidden sockets remain. --- ## Runbook 6 — Credential or input capture **Triggers:** `credential_access` (100033), `input_snooper` (100032) These alerts mean a non-allowlisted process has credential material or keyboard input devices open. They map to Gonzalo-style keylogging and local credential harvesting, but are intentionally behavior-based. **Confirm** - Open the snapshot for the alert and inspect the captured `cmdline`, `exe`, `cwd`, parent process, and fd target. - For `input_snooper`, confirm whether the process is an expected compositor, display server, input remapper, or accessibility tool. Add stable legitimate process names to `input_snooper_allow_comms`. - For `credential_access`, identify which secret was opened: shadow DB, private SSH key, browser credential store, or NetworkManager secret profile. Add only well-understood auth/keyring/browser readers to `credential_access_allow_comms`. **Contain** - Preserve evidence first for unknown processes: copy the executable path if it still exists, capture `/proc//maps`, and export the incident. - Rotate exposed credentials and SSH keys after containment; assume browser session cookies and saved passwords may be compromised if browser stores were opened by an unknown process. --- ## Runbook 7 — Sensor tampering **Triggers:** `fim_modified` on a Sentinel-owned path (self-integrity), a stale or missing heartbeat surfaced by the off-box `watchdog`, or the dashboard going unreachable. Disabling the alarm is step zero of a competent intrusion — so a silent or edited Sentinel is itself an alert. **Confirm** - From the **watchdog host** (never the protected host): ```bash enodia-sentinel watchdog --url https://:8787 --token --max-age 120 ``` Exit 1 means down, unreachable, or stale. - On the host (if you still trust it): `enodia-sentinel status` and check whether Sentinel's own binaries/config/units/hook changed: ```bash enodia-sentinel fim-check # SELF_PATHS are always watched ``` **Contain** - If Sentinel's files were edited, treat the host as compromised: an attacker with the privilege to edit the sensor had the privilege to do far more. - Move to evidence-preservation and offline analysis rather than trusting any further local output. **Recover** - Reinstall Sentinel from a trusted source and re-baseline on a clean host. - Harden against recurrence: `chattr +i` on Sentinel's binaries, config, units, baselines, and the pacman hook; keep the watchdog running off-box. **False positives:** a legitimate Sentinel upgrade changes its own files — but that should coincide with a package transaction. An edit outside one is the suspicious case. --- ## After any incident - Keep the evidence bundle; do not delete snapshots while an investigation is open (retention is bounded by `max_snapshots` / `max_snapshot_age_days`). - Re-baseline (`baseline`, `fim-update`) only once the host is confirmed clean — baselining a compromised host blesses the compromise. - Re-run `enodia-sentinel posture check` to close the hygiene gaps that enabled the intrusion. - The incident-grouping commands (`incident show/export`) on the [roadmap](ROADMAP.md) will eventually automate the evidence-bundle steps above.