enodia-sentinal/docs/RUNBOOKS.md

# Enodia Sentinel Incident Response Runbooks

Practical, conservative response playbooks for the alerts Sentinel raises. Each
runbook follows the same arc: **confirm → preserve → contain → recover**, and
prefers reversible, manual steps. Sentinel does not perform automatic
containment (see [THREAT_MODEL.md](THREAT_MODEL.md)); these are operator actions.

> A note on trust: if you suspect kernel-level compromise, remember that the
> tools you run (`ss`, `ps`, even `ls`) may be lying to you. Cross-check from the
> dashboard on another machine, and when in doubt, capture evidence and analyze
> the disk offline.

## The First Five Minutes

For any alert, before changing anything:

1. **Read the snapshot.** Every alert writes `alert-*.log` (human) and
   `alert-*.json` (machine) under `/var/log/enodia-sentinel`. Note the
   `signature`, `sid`, severity, PIDs, executable paths, and remote peers.
2. **Get host state.** `enodia-sentinel status` — is the daemon healthy, how many
   alerts, when was the last.
3. **Triage known noise.** `enodia-sentinel triage` separates likely false
   positives from findings that need a human.
4. **Build a dry-run response plan.** Find the incident and review the proposed
   actions:

   ```bash
   enodia-sentinel incident list
   enodia-sentinel respond plan <incident-id>
   ```

   The CLI writes a copy under
   `/var/log/enodia-sentinel/response-plans/` and appends
   `/var/log/enodia-sentinel/response-audit.log`, so the reviewed plan can be
   handed off or compared with later actions.

5. **Preserve evidence before you touch anything:**

   ```bash
   tar -C /var/log -czf /tmp/enodia-evidence-$(date +%s).tgz enodia-sentinel
   ```

Then pick the runbook below that matches the signature.

---

## Runbook 1 — Reverse shell / suspicious egress

**Triggers:** `reverse_shell` (sid 100010), `egress` (sid 100016),
`stealth_network` (sid 100036).

An interpreter wired to a socket, an interpreter holding a connection to a
public IP, or activity over raw/SCTP/DCCP/packet/XDP/TIPC/vsock families - the
classic interactive-C2, beaconing, and covert-channel shapes.

**Confirm**

- Open the snapshot; identify the PID, `comm`, parent process, and remote peer.
- Inspect live (these views may be untrustworthy if root is compromised):

  ```bash
  ps -o pid,ppid,user,comm,args -p <pid>
  ls -l /proc/<pid>/fd        # which fds are the socket
  cat /proc/<pid>/environ | tr '\0' '\n'
  ss -tnp | grep <pid>        # the peer
  ```

- Is the peer expected? Compare against `egress_allow_cidrs` in config. A package
  manager or browser to a public IP is normal; `bash`/`python`/`nc` is not.
- For `stealth_network`, check the socket family and owner. Packet/raw sockets
  are normal for approved sniffers and some network managers; SCTP/DCCP/TIPC/XDP
  on a desktop or ordinary server should have a clear service owner. Tune with
  `stealth_network_allow_comms` or, for intentionally used families,
  `stealth_network_allow_kinds`.

**Contain**

- Capture the process before killing it (memory/maps if you have tooling), then:
  `kill -STOP <pid>` to freeze, inspect, and only then `kill -9`.
- Block the peer if it is hostile: `nft add rule inet filter output ip daddr <peer> drop`.
- Find persistence the shell may have dropped — run Runbook 2.

**Recover**

- Rotate any credentials that were reachable from the compromised process.
- Rebuild listener/SUID baselines only after the host is confirmed clean.

**False positives:** legitimate admin scripts, monitoring agents, or P2P clients.
Add trusted egress destinations to `egress_allow_cidrs` rather than disabling the
detector.

---

## Runbook 2 — Persistence tampering

**Triggers:** `persistence` (sid 100015), `fim_modified` on a watched persistence
path.

A change to cron, a systemd unit, an SSH `authorized_keys`, a shell rc file, or
sudo/account config — how an attacker survives a reboot.

**Confirm**

- The snapshot names the changed path. Diff it against the FIM baseline and
  package ownership:

  ```bash
  enodia-sentinel fim-check
  enodia-sentinel fim-check --packages     # does a package own & vouch for it?
  ```

- For SSH keys: is the added key yours? For systemd/cron: does the unit run a
  binary in a writable or unusual location?

**Contain**

- Remove the unauthorized entry (the added key, cron line, or unit). Keep a copy
  in your evidence bundle first.
- `systemctl disable --now <unit>` for a malicious service; check what it spawned.

**Recover**

- After removing persistence, hunt for the process it launched (Runbook 1) and
  any trojaned binaries (Runbook 3).
- Re-anchor the baseline only once clean: `enodia-sentinel fim-update`.

**False positives:** legitimate package upgrades and admin edits. The pacman hook
auto-runs `fim-update` for normal transactions; an unexplained change outside a
transaction is the suspicious case.

---

## Runbook 3 — Trojaned binary / package tampering

**Triggers:** `fim_modified` / `fim_pkg_modified` on a system binary,
`pkgdb_tamper` (package DB edited out of band), `pkg_signature_mismatch`
(on-disk file ≠ signed package), `pacman_siglevel_disabled`.

**Confirm**

- Layer the integrity checks from weakest to strongest trust anchor:

  ```bash
  enodia-sentinel fim-check --packages   # vs FIM baseline + pacman -Qkk
  enodia-sentinel pkgdb-check            # was the local checksum DB rewritten?
  enodia-sentinel pkgdb-verify --sample 200   # vs signed cache packages (.MTREE)
  ```

- `pkg_signature_mismatch` is the strongest signal: it bypasses the mutable local
  DB. Treat a mismatch on a system binary as a trojaned file until proven
  otherwise. `pkgdb_tamper` suggests someone tried to hide exactly this.

**Contain**

- Do not execute the suspect binary. If it is currently running, freeze it
  (Runbook 1).
- Confirm package-signature policy is intact: `enodia-sentinel posture check`
  (flags `pacman_siglevel_disabled`).

**Recover**

- Identify the owning package and reinstall from a trusted source:

  ```bash
  pacman -Qo /path/to/binary
  pacman -S --overwrite '*' <package>     # reinstall the verified file
  enodia-sentinel fim-check --packages    # confirm it now matches
  ```

- Then re-anchor: `enodia-sentinel fim-update` and re-run `pkgdb-check`.

**False positives:** a mid-upgrade snapshot can momentarily disagree. Re-run after
the transaction completes; a persistent mismatch is the real finding.

---

## Runbook 4 — Hidden listener / rootkit

**Triggers:** `rootkit_hidden_process` (100022), `rootkit_hidden_module` (100023),
`rootkit_hidden_port` (100024), `promiscuous_interface` (100025),
`rootkit_known_module` (100028), `rootkit_tainted_module` (100029),
`kernel_tainted` (100030), `rootkit_hidden_udp_port` (100031),
`rootkit_hidden_raw_socket` (100034), `raw_icmp_socket` (100035),
`rootkit_hidden_protocol_socket` (100037), `rootkit_ps_hidden_process` (100038),
and `new_listener` (100013) for an unexplained open port.

These come from cross-view checks: the same question asked two ways, answered
differently, because something is hiding.

**Confirm**

- Re-run the cross-view sweep and capture it:

  ```bash
  enodia-sentinel rootcheck
  ```

- `new_listener`: identify the owning process and whether its binary is
  package-owned. A hidden port that `rootcheck` sees but `ss` does not is far more
  serious than a forgotten service.
- `rootkit_ps_hidden_process`: compare `/proc/<pid>/cmdline`, `/proc/<pid>/exe`,
  and `/proc/<pid>/maps` against `ps`/`pgrep` output. A process visible in
  `/proc` but missing from `ps` strongly suggests userland process-tool hiding.
- `rootkit_hidden_udp_port`: check whether a legitimate UDP service owns the
  port; a port present in `/proc/net/udp*` but absent from `ss -u` suggests tool
  output may be hooked.
- `rootkit_hidden_raw_socket` or `raw_icmp_socket`: look for authorized ping,
  monitoring, traceroute, or packet tooling first. A persistent raw ICMP socket
  with no operator explanation is consistent with ICMP knockers/backdoors and
  should be investigated with process fd tables, package provenance, and memory
  capture before killing the process.
- `rootkit_hidden_protocol_socket`: inspect SCTP, DCCP, packet, TIPC, or XDP
  use. If `/proc/net` sees the family but `ss` does not, treat normal socket
  tooling as suspect and pivot to offline evidence or the dashboard from another
  host.
- A `rootkit_hidden_module` names a live kernel module absent from
  `/proc/modules` — strong evidence of an LKM rootkit.
- A `rootkit_known_module` finding names a module associated with public LKM
  rootkits; treat it as a high-confidence compromise until proven otherwise.
- A `rootkit_tainted_module` or `kernel_tainted` finding may be legitimate
  vendor/DKMS software, but unsigned/out-of-tree or force-loaded modules should
  be explained and allowlisted only after review.

**Contain**

- Assume user-space tools are compromised. Pivot to the read-only dashboard from
  **another machine** to cross-check, and prefer offline analysis.
- Do not `rmmod` blindly — capture the module and system state first.

**Recover**

- A confirmed kernel rootkit means **rebuild from known-good media**; you cannot
  reliably clean a host whose kernel is hooked. Preserve the disk image first.

**False positives:** PID exit races (Sentinel re-verifies hidden PIDs to reduce
these), a legitimately promiscuous interface (a sniffer/monitoring tool you
run), and expected tainted vendor/DKMS modules. Confirm the owning process or
module provenance before escalating; use `rootcheck_module_allow` only for
modules you intentionally trust.

---

## Runbook 5 — Memory obfuscation or process-hiding library

**Triggers:** `memory_obfuscation` (100039), `process_hiding_library` (100048)

These alerts come from `/proc/<pid>/maps`. They do not read process memory; they
flag map shapes associated with encrypted/packed payloads or userland hiding:
RWX pages, executable anonymous memory, executable `memfd`/deleted mappings, or
libraries with process-hiding names such as `libhide`.

**Confirm**

- Open the snapshot and inspect `cmdline`, `exe`, parent process, and captured
  memory maps for the PID.
- For `process_hiding_library`, check whether the mapped library is package-owned
  and whether it is loaded into tools like `ps`, `pgrep`, `ss`, shells, or admin
  utilities.
- For `memory_obfuscation`, distinguish expected JIT runtimes from unknown
  implants. Tune known runtimes with `memory_obfuscation_allow_comms`; use
  `memory_obfuscation_allow_paths` only for well-understood local runtimes or
  instrumentation.

**Contain**

- Preserve the executable and `/proc/<pid>/maps` before killing. If the alert is
  for executable `memfd` or deleted mappings, capture memory first if you have
  trusted tooling; the on-disk payload may be gone.
- Re-run `rootcheck` after containment to confirm no process-tool hiding or
  hidden sockets remain.

---

## Runbook 6 — Credential or input capture

**Triggers:** `credential_access` (100033), `input_snooper` (100032)

These alerts mean a non-allowlisted process has credential material or keyboard
input devices open. They map to Gonzalo-style keylogging and local credential
harvesting, but are intentionally behavior-based.

**Confirm**

- Open the snapshot for the alert and inspect the captured `cmdline`, `exe`,
  `cwd`, parent process, and fd target.
- For `input_snooper`, confirm whether the process is an expected compositor,
  display server, input remapper, or accessibility tool. Add stable legitimate
  process names to `input_snooper_allow_comms`.
- For `credential_access`, identify which secret was opened: shadow DB, private
  SSH key, browser credential store, or NetworkManager secret profile. Add
  only well-understood auth/keyring/browser readers to
  `credential_access_allow_comms`.

**Contain**

- Preserve evidence first for unknown processes: copy the executable path if it
  still exists, capture `/proc/<pid>/maps`, and export the incident.
- Rotate exposed credentials and SSH keys after containment; assume browser
  session cookies and saved passwords may be compromised if browser stores were
  opened by an unknown process.

---

## Runbook 7 — Sensor tampering

**Triggers:** `fim_modified` on a Sentinel-owned path (self-integrity), a stale or
missing heartbeat surfaced by the off-box `watchdog`, or the dashboard going
unreachable.

Disabling the alarm is step zero of a competent intrusion — so a silent or
edited Sentinel is itself an alert.

**Confirm**

- From the **watchdog host** (never the protected host):

  ```bash
  enodia-sentinel watchdog --url https://<host>:8787 --token <token> --max-age 120
  ```

  Exit 1 means down, unreachable, or stale.
- On the host (if you still trust it): `enodia-sentinel status` and check whether
  Sentinel's own binaries/config/units/hook changed:

  ```bash
  enodia-sentinel fim-check        # SELF_PATHS are always watched
  ```

**Contain**

- If Sentinel's files were edited, treat the host as compromised: an attacker with
  the privilege to edit the sensor had the privilege to do far more.
- Move to evidence-preservation and offline analysis rather than trusting any
  further local output.

**Recover**

- Reinstall Sentinel from a trusted source and re-baseline on a clean host.
- Harden against recurrence: `chattr +i` on Sentinel's binaries, config, units,
  baselines, and the pacman hook; keep the watchdog running off-box.

**False positives:** a legitimate Sentinel upgrade changes its own files — but
that should coincide with a package transaction. An edit outside one is the
suspicious case.

---

## After any incident

- Keep the evidence bundle; do not delete snapshots while an investigation is open
  (retention is bounded by `max_snapshots` / `max_snapshot_age_days`).
- Re-baseline (`baseline`, `fim-update`) only once the host is confirmed clean —
  baselining a compromised host blesses the compromise.
- Re-run `enodia-sentinel posture check` to close the hygiene gaps that enabled
  the intrusion.
- The incident-grouping commands (`incident show/export`) on the
  [roadmap](ROADMAP.md) will eventually automate the evidence-bundle steps above.