Add egress drill + sid-verified harness and IR runbooks (roadmap v0.8)

Red-team harness: every detector now has a named drill with an expected
signature/sid, shown by `sentinel-redteam --list`. Add the missing
`egress` drill (the one detector without one) — explicit-only, since it
is the sole drill that emits real outbound traffic. After firing, the
harness runs `enodia-sentinel check` and reports PASS/MISS per poll
detection, turning it into a shared regression check.

Add docs/RUNBOOKS.md: confirm/preserve/contain/recover playbooks for
reverse shells & egress, persistence, trojaned binaries / package
tampering, hidden listeners & rootkits, and sensor tampering — each tied
to the real signatures, sids, and commands. Linked from README and the
OPERATIONS alert workflow.

Both close the remaining smaller v0.8 roadmap items.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

README.md

@@ -36,6 +36,8 @@ Project docs:
   response, fleet, and assurance layers.
 - [Operations guide](docs/OPERATIONS.md) — install checks, health checks, alert
   workflow, baseline hygiene, and evidence export.
+- [IR runbooks](docs/RUNBOOKS.md) — confirm/preserve/contain/recover playbooks
+  per alert: reverse shells, persistence, trojaned binaries, rootkits, tampering.
 
 > **Two implementations, on purpose.** The project began as a bash prototype
 > (`src/sentinel.sh`, kept as the regression **oracle**) and was re-architected

docs/OPERATIONS.md

@@ -96,7 +96,8 @@ When Sentinel fires:
    cp -a /var/log/enodia-sentinel /tmp/enodia-sentinel-evidence
    ```
 
-7. Contain manually for now. Future response commands should produce a dry-run
+7. Contain manually for now, following the matching playbook in
+   [RUNBOOKS.md](RUNBOOKS.md). Future response commands should produce a dry-run
    plan first.
 
 ## Common Findings

docs/ROADMAP.md

@@ -32,10 +32,10 @@ work that is bigger than single alerts.
   package signatures. Still to come: unexpected enabled services and unsafe
   systemd unit settings.
 - ✅ Add a machine-readable `status --json` command for automation.
-- Update the red-team harness so every current detection has a named drill and
+- ✅ Update the red-team harness so every current detection has a named drill and
-  expected `sid`.
+  expected `sid` (`sentinel-redteam --list`), with a PASS/MISS verification pass.
-- Document incident response runbooks for reverse shells, persistence, trojaned
+- ✅ Document incident response runbooks for reverse shells, persistence, trojaned
-  binaries, hidden listeners, and sensor tampering.
+  binaries, hidden listeners, and sensor tampering ([RUNBOOKS.md](RUNBOOKS.md)).
 
 Exit criteria:
 

docs/RUNBOOKS.md

@@ -0,0 +1,250 @@
+# Enodia Sentinel Incident Response Runbooks
+
+Practical, conservative response playbooks for the alerts Sentinel raises. Each
+runbook follows the same arc: **confirm → preserve → contain → recover**, and
+prefers reversible, manual steps. Sentinel does not perform automatic
+containment (see [THREAT_MODEL.md](THREAT_MODEL.md)); these are operator actions.
+
+> A note on trust: if you suspect kernel-level compromise, remember that the
+> tools you run (`ss`, `ps`, even `ls`) may be lying to you. Cross-check from the
+> dashboard on another machine, and when in doubt, capture evidence and analyze
+> the disk offline.
+
+## The First Five Minutes
+
+For any alert, before changing anything:
+
+1. **Read the snapshot.** Every alert writes `alert-*.log` (human) and
+   `alert-*.json` (machine) under `/var/log/enodia-sentinel`. Note the
+   `signature`, `sid`, severity, PIDs, executable paths, and remote peers.
+2. **Get host state.** `enodia-sentinel status` — is the daemon healthy, how many
+   alerts, when was the last.
+3. **Triage known noise.** `enodia-sentinel triage` separates likely false
+   positives from findings that need a human.
+4. **Preserve evidence before you touch anything:**
+
+   ```bash
+   tar -C /var/log -czf /tmp/enodia-evidence-$(date +%s).tgz enodia-sentinel
+   ```
+
+Then pick the runbook below that matches the signature.
+
+---
+
+## Runbook 1 — Reverse shell / suspicious egress
+
+**Triggers:** `reverse_shell` (sid 100010), `egress` (sid 100016).
+
+An interpreter wired to a socket, or an interpreter holding a connection to a
+public IP — the classic interactive-C2 and beaconing shapes.
+
+**Confirm**
+
+- Open the snapshot; identify the PID, `comm`, parent process, and remote peer.
+- Inspect live (these views may be untrustworthy if root is compromised):
+
+  ```bash
+  ps -o pid,ppid,user,comm,args -p <pid>
+  ls -l /proc/<pid>/fd        # which fds are the socket
+  cat /proc/<pid>/environ | tr '\0' '\n'
+  ss -tnp | grep <pid>        # the peer
+  ```
+
+- Is the peer expected? Compare against `egress_allow_cidrs` in config. A package
+  manager or browser to a public IP is normal; `bash`/`python`/`nc` is not.
+
+**Contain**
+
+- Capture the process before killing it (memory/maps if you have tooling), then:
+  `kill -STOP <pid>` to freeze, inspect, and only then `kill -9`.
+- Block the peer if it is hostile: `nft add rule inet filter output ip daddr <peer> drop`.
+- Find persistence the shell may have dropped — run Runbook 2.
+
+**Recover**
+
+- Rotate any credentials that were reachable from the compromised process.
+- Rebuild listener/SUID baselines only after the host is confirmed clean.
+
+**False positives:** legitimate admin scripts, monitoring agents, or P2P clients.
+Add trusted egress destinations to `egress_allow_cidrs` rather than disabling the
+detector.
+
+---
+
+## Runbook 2 — Persistence tampering
+
+**Triggers:** `persistence` (sid 100015), `fim_modified` on a watched persistence
+path.
+
+A change to cron, a systemd unit, an SSH `authorized_keys`, a shell rc file, or
+sudo/account config — how an attacker survives a reboot.
+
+**Confirm**
+
+- The snapshot names the changed path. Diff it against the FIM baseline and
+  package ownership:
+
+  ```bash
+  enodia-sentinel fim-check
+  enodia-sentinel fim-check --packages     # does a package own & vouch for it?
+  ```
+
+- For SSH keys: is the added key yours? For systemd/cron: does the unit run a
+  binary in a writable or unusual location?
+
+**Contain**
+
+- Remove the unauthorized entry (the added key, cron line, or unit). Keep a copy
+  in your evidence bundle first.
+- `systemctl disable --now <unit>` for a malicious service; check what it spawned.
+
+**Recover**
+
+- After removing persistence, hunt for the process it launched (Runbook 1) and
+  any trojaned binaries (Runbook 3).
+- Re-anchor the baseline only once clean: `enodia-sentinel fim-update`.
+
+**False positives:** legitimate package upgrades and admin edits. The pacman hook
+auto-runs `fim-update` for normal transactions; an unexplained change outside a
+transaction is the suspicious case.
+
+---
+
+## Runbook 3 — Trojaned binary / package tampering
+
+**Triggers:** `fim_modified` / `fim_pkg_modified` on a system binary,
+`pkgdb_tamper` (package DB edited out of band), `pkg_signature_mismatch`
+(on-disk file ≠ signed package), `pacman_siglevel_disabled`.
+
+**Confirm**
+
+- Layer the integrity checks from weakest to strongest trust anchor:
+
+  ```bash
+  enodia-sentinel fim-check --packages   # vs FIM baseline + pacman -Qkk
+  enodia-sentinel pkgdb-check            # was the local checksum DB rewritten?
+  enodia-sentinel pkgdb-verify --sample 200   # vs signed cache packages (.MTREE)
+  ```
+
+- `pkg_signature_mismatch` is the strongest signal: it bypasses the mutable local
+  DB. Treat a mismatch on a system binary as a trojaned file until proven
+  otherwise. `pkgdb_tamper` suggests someone tried to hide exactly this.
+
+**Contain**
+
+- Do not execute the suspect binary. If it is currently running, freeze it
+  (Runbook 1).
+- Confirm package-signature policy is intact: `enodia-sentinel posture check`
+  (flags `pacman_siglevel_disabled`).
+
+**Recover**
+
+- Identify the owning package and reinstall from a trusted source:
+
+  ```bash
+  pacman -Qo /path/to/binary
+  pacman -S --overwrite '*' <package>     # reinstall the verified file
+  enodia-sentinel fim-check --packages    # confirm it now matches
+  ```
+
+- Then re-anchor: `enodia-sentinel fim-update` and re-run `pkgdb-check`.
+
+**False positives:** a mid-upgrade snapshot can momentarily disagree. Re-run after
+the transaction completes; a persistent mismatch is the real finding.
+
+---
+
+## Runbook 4 — Hidden listener / rootkit
+
+**Triggers:** `rootkit_hidden_process` (100022), `rootkit_hidden_module` (100023),
+`rootkit_hidden_port` (100024), `promiscuous_interface` (100025), and
+`new_listener` (100013) for an unexplained open port.
+
+These come from cross-view checks: the same question asked two ways, answered
+differently, because something is hiding.
+
+**Confirm**
+
+- Re-run the cross-view sweep and capture it:
+
+  ```bash
+  enodia-sentinel rootcheck
+  ```
+
+- `new_listener`: identify the owning process and whether its binary is
+  package-owned. A hidden port that `rootcheck` sees but `ss` does not is far more
+  serious than a forgotten service.
+- A `rootkit_hidden_module` names a live kernel module absent from
+  `/proc/modules` — strong evidence of an LKM rootkit.
+
+**Contain**
+
+- Assume user-space tools are compromised. Pivot to the read-only dashboard from
+  **another machine** to cross-check, and prefer offline analysis.
+- Do not `rmmod` blindly — capture the module and system state first.
+
+**Recover**
+
+- A confirmed kernel rootkit means **rebuild from known-good media**; you cannot
+  reliably clean a host whose kernel is hooked. Preserve the disk image first.
+
+**False positives:** PID exit races (Sentinel re-verifies hidden PIDs to reduce
+these) and a legitimately promiscuous interface (a sniffer/monitoring tool you
+run). Confirm the owning process before escalating.
+
+---
+
+## Runbook 5 — Sensor tampering
+
+**Triggers:** `fim_modified` on a Sentinel-owned path (self-integrity), a stale or
+missing heartbeat surfaced by the off-box `watchdog`, or the dashboard going
+unreachable.
+
+Disabling the alarm is step zero of a competent intrusion — so a silent or
+edited Sentinel is itself an alert.
+
+**Confirm**
+
+- From the **watchdog host** (never the protected host):
+
+  ```bash
+  enodia-sentinel watchdog --url http://<host>:8787 --token <token> --max-age 120
+  ```
+
+  Exit 1 means down, unreachable, or stale.
+- On the host (if you still trust it): `enodia-sentinel status` and check whether
+  Sentinel's own binaries/config/units/hook changed:
+
+  ```bash
+  enodia-sentinel fim-check        # SELF_PATHS are always watched
+  ```
+
+**Contain**
+
+- If Sentinel's files were edited, treat the host as compromised: an attacker with
+  the privilege to edit the sensor had the privilege to do far more.
+- Move to evidence-preservation and offline analysis rather than trusting any
+  further local output.
+
+**Recover**
+
+- Reinstall Sentinel from a trusted source and re-baseline on a clean host.
+- Harden against recurrence: `chattr +i` on Sentinel's binaries, config, units,
+  baselines, and the pacman hook; keep the watchdog running off-box.
+
+**False positives:** a legitimate Sentinel upgrade changes its own files — but
+that should coincide with a package transaction. An edit outside one is the
+suspicious case.
+
+---
+
+## After any incident
+
+- Keep the evidence bundle; do not delete snapshots while an investigation is open
+  (retention is bounded by `max_snapshots` / `max_snapshot_age_days`).
+- Re-baseline (`baseline`, `fim-update`) only once the host is confirmed clean —
+  baselining a compromised host blesses the compromise.
+- Re-run `enodia-sentinel posture check` to close the hygiene gaps that enabled
+  the intrusion.
+- The incident-grouping commands (`incident show/export`) on the
+  [roadmap](ROADMAP.md) will eventually automate the evidence-bundle steps above.

src/sentinel-redteam

@@ -8,11 +8,13 @@
 # everything is cleaned up on exit.
 #
 # Usage:
-#   sentinel-redteam            run all drills (default 20s each window)
+#   sentinel-redteam            run the default drills (20s each window)
 #   sentinel-redteam <name>...  run only the named drills
-#   sentinel-redteam --list     list available drills
+#   sentinel-redteam --list     list drills with their expected detector + sid
 #
-# Drills: reverse_shell ld_preload deleted_exe new_listener new_suid persistence
+# Every current detector has a named drill with an expected signature/sid (see
+# --list), so this doubles as a shared regression suite: after firing, it runs
+# `enodia-sentinel check` and reports PASS/MISS for each poll detection.
 #
 # Watch it work, in another terminal:
 #   sudo tail -f /var/log/enodia-sentinel/events.log
@@ -25,6 +27,24 @@ TMP="$(mktemp -d /tmp/enodia-drill.XXXXXX)"
 PIDS=()
 SANDBOX="$TMP/persistence-sandbox"        # used by the persistence drill
 LISTEN_PORT=14444
+# egress drill target — the ONE drill that makes a real outbound connection.
+EGRESS_HOST="${EGRESS_HOST:-1.1.1.1}"
+EGRESS_PORT="${EGRESS_PORT:-53}"
+
+# Drill metadata: name -> "signature sid kind", where kind is how it's verified:
+#   poll   = the one-shot `check` should catch it
+#   event  = only the eBPF monitor sees it (vanishes between polls)
+#   cond   = conditional (needs extra setup; see the drill's note)
+declare -A DRILL_SIG=(
+    [reverse_shell]="reverse_shell 100010 poll"
+    [ld_preload]="ld_preload 100011 poll"
+    [deleted_exe]="deleted_exe 100012 poll"
+    [new_listener]="new_listener 100013 poll"
+    [new_suid]="new_suid 100014 poll"
+    [persistence]="persistence 100015 cond"
+    [egress]="egress 100016 poll"
+    [ebpf_exec]="exec_anomaly 100001/100002 event"
+)
 
 cleanup() {
     echo "[*] cleaning up drill artifacts..."
@@ -110,6 +130,22 @@ drill_persistence() {
     echo "    -> wrote $SANDBOX/authorized_keys"
 }
 
+drill_egress() {
+    note egress "an interpreter (python) holding an outbound TCP connection to a public IP"
+    echo "    WARNING: unlike the other drills, this makes ONE real outbound TCP"
+    echo "    connection to a public host (${EGRESS_HOST}:${EGRESS_PORT}). No data is sent;"
+    echo "    only the handshake. Set EGRESS_HOST/EGRESS_PORT to retarget, or skip it."
+    # python3 is an 'interpreter' to the egress detector; an established socket
+    # to a globally-routable IP is exactly the C2/exfil shape it flags (sid 100016).
+    python3 -c 'import socket,sys,time
+s=socket.socket()
+try:
+    s.connect((sys.argv[1], int(sys.argv[2]))); time.sleep(int(sys.argv[3]))
+except Exception: pass' "$EGRESS_HOST" "$EGRESS_PORT" "$HOLD" >/dev/null 2>&1 &
+    PIDS+=("$!")
+    echo "    -> python3 (pid $!) connected to ${EGRESS_HOST}:${EGRESS_PORT} for ${HOLD}s"
+}
+
 drill_ebpf_exec() {
     note ebpf_exec "short-lived /tmp exec + reverse-shell-pattern argv (only the eBPF monitor catches these)"
     cp /bin/true "$TMP/enodia-drill-exec" 2>/dev/null
@@ -130,14 +166,29 @@ run() {
         new_listener)  drill_new_listener ;;
         new_suid)      drill_new_suid ;;
         persistence)   drill_persistence ;;
+        egress)        drill_egress ;;
         ebpf_exec)     drill_ebpf_exec ;;
         *) echo "unknown drill: $1" >&2; return 1 ;;
     esac
 }
 
+# egress is excluded from the default run because it is the only drill that
+# emits real network traffic; name it explicitly to include it.
 ALL=(reverse_shell ld_preload deleted_exe new_listener new_suid persistence ebpf_exec)
+KNOWN=(reverse_shell ld_preload deleted_exe new_listener new_suid persistence egress ebpf_exec)
 
-if [ "${1:-}" = "--list" ]; then printf '%s\n' "${ALL[@]}"; exit 0; fi
+if [ "${1:-}" = "--list" ]; then
+    printf '%-15s %-16s %-7s %s\n' DRILL SIGNATURE SID KIND
+    for d in "${KNOWN[@]}"; do
+        read -r sig sid kind <<<"${DRILL_SIG[$d]}"
+        printf '%-15s %-16s %-7s %s\n' "$d" "$sig" "$sid" "$kind"
+    done
+    echo
+    echo "kinds: poll=caught by 'check'; event=eBPF-only; cond=needs extra setup"
+    echo "note:  rootcheck (100022-25) and posture (100040-47) are verified by"
+    echo "       'enodia-sentinel rootcheck' / 'posture check', not attack drills."
+    exit 0
+fi
 
 echo "######################################################################"
 echo "#  ENODIA SENTINEL — RED TEAM DRILL                                  #"
@@ -148,14 +199,31 @@ echo "######################################################################"
 if [ "$#" -gt 0 ]; then targets=("$@"); else targets=("${ALL[@]}"); fi
 for d in "${targets[@]}"; do run "$d"; done
 
-echo -e "\n[*] drills live for ${HOLD}s. Run a one-shot poll check now:"
+# --- verification: run one poll check and confirm each expected signature ----
+echo -e "\n[*] drills live for ${HOLD}s. Verifying with a one-shot poll check..."
 if command -v enodia-sentinel >/dev/null 2>&1; then
-    enodia-sentinel check 2>/dev/null | grep -i 'drill\|/tmp/enodia' \
+    sleep 1
-        && echo "    ^ Sentinel's poll detectors saw the drills." \
+    OUT="$(enodia-sentinel check 2>/dev/null)"
-        || echo "    (no poll hits yet — watch /var/log/enodia-sentinel/events.log)"
+    pass=0; miss=0
+    for d in "${targets[@]}"; do
+        read -r sig sid kind <<<"${DRILL_SIG[$d]}"
+        case "$kind" in
+            poll)
+                if grep -qw "$sig" <<<"$OUT"; then
+                    printf '    PASS  %-15s %s (sid %s)\n' "$d" "$sig" "$sid"; pass=$((pass+1))
+                else
+                    printf '    MISS  %-15s expected %s (sid %s)\n' "$d" "$sig" "$sid"; miss=$((miss+1))
+                fi ;;
+            event)
+                printf '    INFO  %-15s %s — eBPF-only; see events.log (sid %s)\n' "$d" "$sig" "$sid" ;;
+            cond)
+                printf '    INFO  %-15s %s — conditional; see the drill note (sid %s)\n' "$d" "$sig" "$sid" ;;
+        esac
+    done
+    echo "    ---- ${pass} passed, ${miss} missed (poll detections) ----"
+    [ "$miss" -gt 0 ] && echo "    MISS can be a real regression OR a timing/baseline gap — re-check events.log."
 else
     echo "    (enodia-sentinel not on PATH — watch /var/log/enodia-sentinel/events.log)"
 fi
-echo "    The 'ebpf_exec' drill only shows up with the eBPF monitor (service as root)."
 echo "[*] holding for ${HOLD}s, then cleaning up..."
 sleep "$HOLD"