diff --git a/README.md b/README.md index 33c2754..af00d81 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,8 @@ Project docs: response, fleet, and assurance layers. - [Operations guide](docs/OPERATIONS.md) — install checks, health checks, alert workflow, baseline hygiene, and evidence export. +- [IR runbooks](docs/RUNBOOKS.md) — confirm/preserve/contain/recover playbooks + per alert: reverse shells, persistence, trojaned binaries, rootkits, tampering. > **Two implementations, on purpose.** The project began as a bash prototype > (`src/sentinel.sh`, kept as the regression **oracle**) and was re-architected diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 0093f5f..e2676e3 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -96,7 +96,8 @@ When Sentinel fires: cp -a /var/log/enodia-sentinel /tmp/enodia-sentinel-evidence ``` -7. Contain manually for now. Future response commands should produce a dry-run +7. Contain manually for now, following the matching playbook in + [RUNBOOKS.md](RUNBOOKS.md). Future response commands should produce a dry-run plan first. ## Common Findings diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 84556aa..b2ab992 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -32,10 +32,10 @@ work that is bigger than single alerts. package signatures. Still to come: unexpected enabled services and unsafe systemd unit settings. - ✅ Add a machine-readable `status --json` command for automation. -- Update the red-team harness so every current detection has a named drill and - expected `sid`. -- Document incident response runbooks for reverse shells, persistence, trojaned - binaries, hidden listeners, and sensor tampering. +- ✅ Update the red-team harness so every current detection has a named drill and + expected `sid` (`sentinel-redteam --list`), with a PASS/MISS verification pass. +- ✅ Document incident response runbooks for reverse shells, persistence, trojaned + binaries, hidden listeners, and sensor tampering ([RUNBOOKS.md](RUNBOOKS.md)). Exit criteria: diff --git a/docs/RUNBOOKS.md b/docs/RUNBOOKS.md new file mode 100644 index 0000000..aebf3d1 --- /dev/null +++ b/docs/RUNBOOKS.md @@ -0,0 +1,250 @@ +# Enodia Sentinel Incident Response Runbooks + +Practical, conservative response playbooks for the alerts Sentinel raises. Each +runbook follows the same arc: **confirm → preserve → contain → recover**, and +prefers reversible, manual steps. Sentinel does not perform automatic +containment (see [THREAT_MODEL.md](THREAT_MODEL.md)); these are operator actions. + +> A note on trust: if you suspect kernel-level compromise, remember that the +> tools you run (`ss`, `ps`, even `ls`) may be lying to you. Cross-check from the +> dashboard on another machine, and when in doubt, capture evidence and analyze +> the disk offline. + +## The First Five Minutes + +For any alert, before changing anything: + +1. **Read the snapshot.** Every alert writes `alert-*.log` (human) and + `alert-*.json` (machine) under `/var/log/enodia-sentinel`. Note the + `signature`, `sid`, severity, PIDs, executable paths, and remote peers. +2. **Get host state.** `enodia-sentinel status` — is the daemon healthy, how many + alerts, when was the last. +3. **Triage known noise.** `enodia-sentinel triage` separates likely false + positives from findings that need a human. +4. **Preserve evidence before you touch anything:** + + ```bash + tar -C /var/log -czf /tmp/enodia-evidence-$(date +%s).tgz enodia-sentinel + ``` + +Then pick the runbook below that matches the signature. + +--- + +## Runbook 1 — Reverse shell / suspicious egress + +**Triggers:** `reverse_shell` (sid 100010), `egress` (sid 100016). + +An interpreter wired to a socket, or an interpreter holding a connection to a +public IP — the classic interactive-C2 and beaconing shapes. + +**Confirm** + +- Open the snapshot; identify the PID, `comm`, parent process, and remote peer. +- Inspect live (these views may be untrustworthy if root is compromised): + + ```bash + ps -o pid,ppid,user,comm,args -p + ls -l /proc//fd # which fds are the socket + cat /proc//environ | tr '\0' '\n' + ss -tnp | grep # the peer + ``` + +- Is the peer expected? Compare against `egress_allow_cidrs` in config. A package + manager or browser to a public IP is normal; `bash`/`python`/`nc` is not. + +**Contain** + +- Capture the process before killing it (memory/maps if you have tooling), then: + `kill -STOP ` to freeze, inspect, and only then `kill -9`. +- Block the peer if it is hostile: `nft add rule inet filter output ip daddr drop`. +- Find persistence the shell may have dropped — run Runbook 2. + +**Recover** + +- Rotate any credentials that were reachable from the compromised process. +- Rebuild listener/SUID baselines only after the host is confirmed clean. + +**False positives:** legitimate admin scripts, monitoring agents, or P2P clients. +Add trusted egress destinations to `egress_allow_cidrs` rather than disabling the +detector. + +--- + +## Runbook 2 — Persistence tampering + +**Triggers:** `persistence` (sid 100015), `fim_modified` on a watched persistence +path. + +A change to cron, a systemd unit, an SSH `authorized_keys`, a shell rc file, or +sudo/account config — how an attacker survives a reboot. + +**Confirm** + +- The snapshot names the changed path. Diff it against the FIM baseline and + package ownership: + + ```bash + enodia-sentinel fim-check + enodia-sentinel fim-check --packages # does a package own & vouch for it? + ``` + +- For SSH keys: is the added key yours? For systemd/cron: does the unit run a + binary in a writable or unusual location? + +**Contain** + +- Remove the unauthorized entry (the added key, cron line, or unit). Keep a copy + in your evidence bundle first. +- `systemctl disable --now ` for a malicious service; check what it spawned. + +**Recover** + +- After removing persistence, hunt for the process it launched (Runbook 1) and + any trojaned binaries (Runbook 3). +- Re-anchor the baseline only once clean: `enodia-sentinel fim-update`. + +**False positives:** legitimate package upgrades and admin edits. The pacman hook +auto-runs `fim-update` for normal transactions; an unexplained change outside a +transaction is the suspicious case. + +--- + +## Runbook 3 — Trojaned binary / package tampering + +**Triggers:** `fim_modified` / `fim_pkg_modified` on a system binary, +`pkgdb_tamper` (package DB edited out of band), `pkg_signature_mismatch` +(on-disk file ≠ signed package), `pacman_siglevel_disabled`. + +**Confirm** + +- Layer the integrity checks from weakest to strongest trust anchor: + + ```bash + enodia-sentinel fim-check --packages # vs FIM baseline + pacman -Qkk + enodia-sentinel pkgdb-check # was the local checksum DB rewritten? + enodia-sentinel pkgdb-verify --sample 200 # vs signed cache packages (.MTREE) + ``` + +- `pkg_signature_mismatch` is the strongest signal: it bypasses the mutable local + DB. Treat a mismatch on a system binary as a trojaned file until proven + otherwise. `pkgdb_tamper` suggests someone tried to hide exactly this. + +**Contain** + +- Do not execute the suspect binary. If it is currently running, freeze it + (Runbook 1). +- Confirm package-signature policy is intact: `enodia-sentinel posture check` + (flags `pacman_siglevel_disabled`). + +**Recover** + +- Identify the owning package and reinstall from a trusted source: + + ```bash + pacman -Qo /path/to/binary + pacman -S --overwrite '*' # reinstall the verified file + enodia-sentinel fim-check --packages # confirm it now matches + ``` + +- Then re-anchor: `enodia-sentinel fim-update` and re-run `pkgdb-check`. + +**False positives:** a mid-upgrade snapshot can momentarily disagree. Re-run after +the transaction completes; a persistent mismatch is the real finding. + +--- + +## Runbook 4 — Hidden listener / rootkit + +**Triggers:** `rootkit_hidden_process` (100022), `rootkit_hidden_module` (100023), +`rootkit_hidden_port` (100024), `promiscuous_interface` (100025), and +`new_listener` (100013) for an unexplained open port. + +These come from cross-view checks: the same question asked two ways, answered +differently, because something is hiding. + +**Confirm** + +- Re-run the cross-view sweep and capture it: + + ```bash + enodia-sentinel rootcheck + ``` + +- `new_listener`: identify the owning process and whether its binary is + package-owned. A hidden port that `rootcheck` sees but `ss` does not is far more + serious than a forgotten service. +- A `rootkit_hidden_module` names a live kernel module absent from + `/proc/modules` — strong evidence of an LKM rootkit. + +**Contain** + +- Assume user-space tools are compromised. Pivot to the read-only dashboard from + **another machine** to cross-check, and prefer offline analysis. +- Do not `rmmod` blindly — capture the module and system state first. + +**Recover** + +- A confirmed kernel rootkit means **rebuild from known-good media**; you cannot + reliably clean a host whose kernel is hooked. Preserve the disk image first. + +**False positives:** PID exit races (Sentinel re-verifies hidden PIDs to reduce +these) and a legitimately promiscuous interface (a sniffer/monitoring tool you +run). Confirm the owning process before escalating. + +--- + +## Runbook 5 — Sensor tampering + +**Triggers:** `fim_modified` on a Sentinel-owned path (self-integrity), a stale or +missing heartbeat surfaced by the off-box `watchdog`, or the dashboard going +unreachable. + +Disabling the alarm is step zero of a competent intrusion — so a silent or +edited Sentinel is itself an alert. + +**Confirm** + +- From the **watchdog host** (never the protected host): + + ```bash + enodia-sentinel watchdog --url http://:8787 --token --max-age 120 + ``` + + Exit 1 means down, unreachable, or stale. +- On the host (if you still trust it): `enodia-sentinel status` and check whether + Sentinel's own binaries/config/units/hook changed: + + ```bash + enodia-sentinel fim-check # SELF_PATHS are always watched + ``` + +**Contain** + +- If Sentinel's files were edited, treat the host as compromised: an attacker with + the privilege to edit the sensor had the privilege to do far more. +- Move to evidence-preservation and offline analysis rather than trusting any + further local output. + +**Recover** + +- Reinstall Sentinel from a trusted source and re-baseline on a clean host. +- Harden against recurrence: `chattr +i` on Sentinel's binaries, config, units, + baselines, and the pacman hook; keep the watchdog running off-box. + +**False positives:** a legitimate Sentinel upgrade changes its own files — but +that should coincide with a package transaction. An edit outside one is the +suspicious case. + +--- + +## After any incident + +- Keep the evidence bundle; do not delete snapshots while an investigation is open + (retention is bounded by `max_snapshots` / `max_snapshot_age_days`). +- Re-baseline (`baseline`, `fim-update`) only once the host is confirmed clean — + baselining a compromised host blesses the compromise. +- Re-run `enodia-sentinel posture check` to close the hygiene gaps that enabled + the intrusion. +- The incident-grouping commands (`incident show/export`) on the + [roadmap](ROADMAP.md) will eventually automate the evidence-bundle steps above. diff --git a/src/sentinel-redteam b/src/sentinel-redteam index 6c33128..65cf0fd 100755 --- a/src/sentinel-redteam +++ b/src/sentinel-redteam @@ -8,11 +8,13 @@ # everything is cleaned up on exit. # # Usage: -# sentinel-redteam run all drills (default 20s each window) +# sentinel-redteam run the default drills (20s each window) # sentinel-redteam ... run only the named drills -# sentinel-redteam --list list available drills +# sentinel-redteam --list list drills with their expected detector + sid # -# Drills: reverse_shell ld_preload deleted_exe new_listener new_suid persistence +# Every current detector has a named drill with an expected signature/sid (see +# --list), so this doubles as a shared regression suite: after firing, it runs +# `enodia-sentinel check` and reports PASS/MISS for each poll detection. # # Watch it work, in another terminal: # sudo tail -f /var/log/enodia-sentinel/events.log @@ -25,6 +27,24 @@ TMP="$(mktemp -d /tmp/enodia-drill.XXXXXX)" PIDS=() SANDBOX="$TMP/persistence-sandbox" # used by the persistence drill LISTEN_PORT=14444 +# egress drill target — the ONE drill that makes a real outbound connection. +EGRESS_HOST="${EGRESS_HOST:-1.1.1.1}" +EGRESS_PORT="${EGRESS_PORT:-53}" + +# Drill metadata: name -> "signature sid kind", where kind is how it's verified: +# poll = the one-shot `check` should catch it +# event = only the eBPF monitor sees it (vanishes between polls) +# cond = conditional (needs extra setup; see the drill's note) +declare -A DRILL_SIG=( + [reverse_shell]="reverse_shell 100010 poll" + [ld_preload]="ld_preload 100011 poll" + [deleted_exe]="deleted_exe 100012 poll" + [new_listener]="new_listener 100013 poll" + [new_suid]="new_suid 100014 poll" + [persistence]="persistence 100015 cond" + [egress]="egress 100016 poll" + [ebpf_exec]="exec_anomaly 100001/100002 event" +) cleanup() { echo "[*] cleaning up drill artifacts..." @@ -110,6 +130,22 @@ drill_persistence() { echo " -> wrote $SANDBOX/authorized_keys" } +drill_egress() { + note egress "an interpreter (python) holding an outbound TCP connection to a public IP" + echo " WARNING: unlike the other drills, this makes ONE real outbound TCP" + echo " connection to a public host (${EGRESS_HOST}:${EGRESS_PORT}). No data is sent;" + echo " only the handshake. Set EGRESS_HOST/EGRESS_PORT to retarget, or skip it." + # python3 is an 'interpreter' to the egress detector; an established socket + # to a globally-routable IP is exactly the C2/exfil shape it flags (sid 100016). + python3 -c 'import socket,sys,time +s=socket.socket() +try: + s.connect((sys.argv[1], int(sys.argv[2]))); time.sleep(int(sys.argv[3])) +except Exception: pass' "$EGRESS_HOST" "$EGRESS_PORT" "$HOLD" >/dev/null 2>&1 & + PIDS+=("$!") + echo " -> python3 (pid $!) connected to ${EGRESS_HOST}:${EGRESS_PORT} for ${HOLD}s" +} + drill_ebpf_exec() { note ebpf_exec "short-lived /tmp exec + reverse-shell-pattern argv (only the eBPF monitor catches these)" cp /bin/true "$TMP/enodia-drill-exec" 2>/dev/null @@ -130,14 +166,29 @@ run() { new_listener) drill_new_listener ;; new_suid) drill_new_suid ;; persistence) drill_persistence ;; + egress) drill_egress ;; ebpf_exec) drill_ebpf_exec ;; *) echo "unknown drill: $1" >&2; return 1 ;; esac } +# egress is excluded from the default run because it is the only drill that +# emits real network traffic; name it explicitly to include it. ALL=(reverse_shell ld_preload deleted_exe new_listener new_suid persistence ebpf_exec) +KNOWN=(reverse_shell ld_preload deleted_exe new_listener new_suid persistence egress ebpf_exec) -if [ "${1:-}" = "--list" ]; then printf '%s\n' "${ALL[@]}"; exit 0; fi +if [ "${1:-}" = "--list" ]; then + printf '%-15s %-16s %-7s %s\n' DRILL SIGNATURE SID KIND + for d in "${KNOWN[@]}"; do + read -r sig sid kind <<<"${DRILL_SIG[$d]}" + printf '%-15s %-16s %-7s %s\n' "$d" "$sig" "$sid" "$kind" + done + echo + echo "kinds: poll=caught by 'check'; event=eBPF-only; cond=needs extra setup" + echo "note: rootcheck (100022-25) and posture (100040-47) are verified by" + echo " 'enodia-sentinel rootcheck' / 'posture check', not attack drills." + exit 0 +fi echo "######################################################################" echo "# ENODIA SENTINEL — RED TEAM DRILL #" @@ -148,14 +199,31 @@ echo "######################################################################" if [ "$#" -gt 0 ]; then targets=("$@"); else targets=("${ALL[@]}"); fi for d in "${targets[@]}"; do run "$d"; done -echo -e "\n[*] drills live for ${HOLD}s. Run a one-shot poll check now:" +# --- verification: run one poll check and confirm each expected signature ---- +echo -e "\n[*] drills live for ${HOLD}s. Verifying with a one-shot poll check..." if command -v enodia-sentinel >/dev/null 2>&1; then - enodia-sentinel check 2>/dev/null | grep -i 'drill\|/tmp/enodia' \ - && echo " ^ Sentinel's poll detectors saw the drills." \ - || echo " (no poll hits yet — watch /var/log/enodia-sentinel/events.log)" + sleep 1 + OUT="$(enodia-sentinel check 2>/dev/null)" + pass=0; miss=0 + for d in "${targets[@]}"; do + read -r sig sid kind <<<"${DRILL_SIG[$d]}" + case "$kind" in + poll) + if grep -qw "$sig" <<<"$OUT"; then + printf ' PASS %-15s %s (sid %s)\n' "$d" "$sig" "$sid"; pass=$((pass+1)) + else + printf ' MISS %-15s expected %s (sid %s)\n' "$d" "$sig" "$sid"; miss=$((miss+1)) + fi ;; + event) + printf ' INFO %-15s %s — eBPF-only; see events.log (sid %s)\n' "$d" "$sig" "$sid" ;; + cond) + printf ' INFO %-15s %s — conditional; see the drill note (sid %s)\n' "$d" "$sig" "$sid" ;; + esac + done + echo " ---- ${pass} passed, ${miss} missed (poll detections) ----" + [ "$miss" -gt 0 ] && echo " MISS can be a real regression OR a timing/baseline gap — re-check events.log." else echo " (enodia-sentinel not on PATH — watch /var/log/enodia-sentinel/events.log)" fi -echo " The 'ebpf_exec' drill only shows up with the eBPF monitor (service as root)." echo "[*] holding for ${HOLD}s, then cleaning up..." sleep "$HOLD"