Add egress drill + sid-verified harness and IR runbooks (roadmap v0.8)
Red-team harness: every detector now has a named drill with an expected signature/sid, shown by `sentinel-redteam --list`. Add the missing `egress` drill (the one detector without one) — explicit-only, since it is the sole drill that emits real outbound traffic. After firing, the harness runs `enodia-sentinel check` and reports PASS/MISS per poll detection, turning it into a shared regression check. Add docs/RUNBOOKS.md: confirm/preserve/contain/recover playbooks for reverse shells & egress, persistence, trojaned binaries / package tampering, hidden listeners & rootkits, and sensor tampering — each tied to the real signatures, sids, and commands. Linked from README and the OPERATIONS alert workflow. Both close the remaining smaller v0.8 roadmap items. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Luna
parent
f2d896a2d7
commit
4015ec872b
5 changed files with 335 additions and 14 deletions
250
docs/RUNBOOKS.md
Normal file
250
docs/RUNBOOKS.md
Normal file
|
|
@ -0,0 +1,250 @@
|
|||
# Enodia Sentinel Incident Response Runbooks
|
||||
|
||||
Practical, conservative response playbooks for the alerts Sentinel raises. Each
|
||||
runbook follows the same arc: **confirm → preserve → contain → recover**, and
|
||||
prefers reversible, manual steps. Sentinel does not perform automatic
|
||||
containment (see [THREAT_MODEL.md](THREAT_MODEL.md)); these are operator actions.
|
||||
|
||||
> A note on trust: if you suspect kernel-level compromise, remember that the
|
||||
> tools you run (`ss`, `ps`, even `ls`) may be lying to you. Cross-check from the
|
||||
> dashboard on another machine, and when in doubt, capture evidence and analyze
|
||||
> the disk offline.
|
||||
|
||||
## The First Five Minutes
|
||||
|
||||
For any alert, before changing anything:
|
||||
|
||||
1. **Read the snapshot.** Every alert writes `alert-*.log` (human) and
|
||||
`alert-*.json` (machine) under `/var/log/enodia-sentinel`. Note the
|
||||
`signature`, `sid`, severity, PIDs, executable paths, and remote peers.
|
||||
2. **Get host state.** `enodia-sentinel status` — is the daemon healthy, how many
|
||||
alerts, when was the last.
|
||||
3. **Triage known noise.** `enodia-sentinel triage` separates likely false
|
||||
positives from findings that need a human.
|
||||
4. **Preserve evidence before you touch anything:**
|
||||
|
||||
```bash
|
||||
tar -C /var/log -czf /tmp/enodia-evidence-$(date +%s).tgz enodia-sentinel
|
||||
```
|
||||
|
||||
Then pick the runbook below that matches the signature.
|
||||
|
||||
---
|
||||
|
||||
## Runbook 1 — Reverse shell / suspicious egress
|
||||
|
||||
**Triggers:** `reverse_shell` (sid 100010), `egress` (sid 100016).
|
||||
|
||||
An interpreter wired to a socket, or an interpreter holding a connection to a
|
||||
public IP — the classic interactive-C2 and beaconing shapes.
|
||||
|
||||
**Confirm**
|
||||
|
||||
- Open the snapshot; identify the PID, `comm`, parent process, and remote peer.
|
||||
- Inspect live (these views may be untrustworthy if root is compromised):
|
||||
|
||||
```bash
|
||||
ps -o pid,ppid,user,comm,args -p <pid>
|
||||
ls -l /proc/<pid>/fd # which fds are the socket
|
||||
cat /proc/<pid>/environ | tr '\0' '\n'
|
||||
ss -tnp | grep <pid> # the peer
|
||||
```
|
||||
|
||||
- Is the peer expected? Compare against `egress_allow_cidrs` in config. A package
|
||||
manager or browser to a public IP is normal; `bash`/`python`/`nc` is not.
|
||||
|
||||
**Contain**
|
||||
|
||||
- Capture the process before killing it (memory/maps if you have tooling), then:
|
||||
`kill -STOP <pid>` to freeze, inspect, and only then `kill -9`.
|
||||
- Block the peer if it is hostile: `nft add rule inet filter output ip daddr <peer> drop`.
|
||||
- Find persistence the shell may have dropped — run Runbook 2.
|
||||
|
||||
**Recover**
|
||||
|
||||
- Rotate any credentials that were reachable from the compromised process.
|
||||
- Rebuild listener/SUID baselines only after the host is confirmed clean.
|
||||
|
||||
**False positives:** legitimate admin scripts, monitoring agents, or P2P clients.
|
||||
Add trusted egress destinations to `egress_allow_cidrs` rather than disabling the
|
||||
detector.
|
||||
|
||||
---
|
||||
|
||||
## Runbook 2 — Persistence tampering
|
||||
|
||||
**Triggers:** `persistence` (sid 100015), `fim_modified` on a watched persistence
|
||||
path.
|
||||
|
||||
A change to cron, a systemd unit, an SSH `authorized_keys`, a shell rc file, or
|
||||
sudo/account config — how an attacker survives a reboot.
|
||||
|
||||
**Confirm**
|
||||
|
||||
- The snapshot names the changed path. Diff it against the FIM baseline and
|
||||
package ownership:
|
||||
|
||||
```bash
|
||||
enodia-sentinel fim-check
|
||||
enodia-sentinel fim-check --packages # does a package own & vouch for it?
|
||||
```
|
||||
|
||||
- For SSH keys: is the added key yours? For systemd/cron: does the unit run a
|
||||
binary in a writable or unusual location?
|
||||
|
||||
**Contain**
|
||||
|
||||
- Remove the unauthorized entry (the added key, cron line, or unit). Keep a copy
|
||||
in your evidence bundle first.
|
||||
- `systemctl disable --now <unit>` for a malicious service; check what it spawned.
|
||||
|
||||
**Recover**
|
||||
|
||||
- After removing persistence, hunt for the process it launched (Runbook 1) and
|
||||
any trojaned binaries (Runbook 3).
|
||||
- Re-anchor the baseline only once clean: `enodia-sentinel fim-update`.
|
||||
|
||||
**False positives:** legitimate package upgrades and admin edits. The pacman hook
|
||||
auto-runs `fim-update` for normal transactions; an unexplained change outside a
|
||||
transaction is the suspicious case.
|
||||
|
||||
---
|
||||
|
||||
## Runbook 3 — Trojaned binary / package tampering
|
||||
|
||||
**Triggers:** `fim_modified` / `fim_pkg_modified` on a system binary,
|
||||
`pkgdb_tamper` (package DB edited out of band), `pkg_signature_mismatch`
|
||||
(on-disk file ≠ signed package), `pacman_siglevel_disabled`.
|
||||
|
||||
**Confirm**
|
||||
|
||||
- Layer the integrity checks from weakest to strongest trust anchor:
|
||||
|
||||
```bash
|
||||
enodia-sentinel fim-check --packages # vs FIM baseline + pacman -Qkk
|
||||
enodia-sentinel pkgdb-check # was the local checksum DB rewritten?
|
||||
enodia-sentinel pkgdb-verify --sample 200 # vs signed cache packages (.MTREE)
|
||||
```
|
||||
|
||||
- `pkg_signature_mismatch` is the strongest signal: it bypasses the mutable local
|
||||
DB. Treat a mismatch on a system binary as a trojaned file until proven
|
||||
otherwise. `pkgdb_tamper` suggests someone tried to hide exactly this.
|
||||
|
||||
**Contain**
|
||||
|
||||
- Do not execute the suspect binary. If it is currently running, freeze it
|
||||
(Runbook 1).
|
||||
- Confirm package-signature policy is intact: `enodia-sentinel posture check`
|
||||
(flags `pacman_siglevel_disabled`).
|
||||
|
||||
**Recover**
|
||||
|
||||
- Identify the owning package and reinstall from a trusted source:
|
||||
|
||||
```bash
|
||||
pacman -Qo /path/to/binary
|
||||
pacman -S --overwrite '*' <package> # reinstall the verified file
|
||||
enodia-sentinel fim-check --packages # confirm it now matches
|
||||
```
|
||||
|
||||
- Then re-anchor: `enodia-sentinel fim-update` and re-run `pkgdb-check`.
|
||||
|
||||
**False positives:** a mid-upgrade snapshot can momentarily disagree. Re-run after
|
||||
the transaction completes; a persistent mismatch is the real finding.
|
||||
|
||||
---
|
||||
|
||||
## Runbook 4 — Hidden listener / rootkit
|
||||
|
||||
**Triggers:** `rootkit_hidden_process` (100022), `rootkit_hidden_module` (100023),
|
||||
`rootkit_hidden_port` (100024), `promiscuous_interface` (100025), and
|
||||
`new_listener` (100013) for an unexplained open port.
|
||||
|
||||
These come from cross-view checks: the same question asked two ways, answered
|
||||
differently, because something is hiding.
|
||||
|
||||
**Confirm**
|
||||
|
||||
- Re-run the cross-view sweep and capture it:
|
||||
|
||||
```bash
|
||||
enodia-sentinel rootcheck
|
||||
```
|
||||
|
||||
- `new_listener`: identify the owning process and whether its binary is
|
||||
package-owned. A hidden port that `rootcheck` sees but `ss` does not is far more
|
||||
serious than a forgotten service.
|
||||
- A `rootkit_hidden_module` names a live kernel module absent from
|
||||
`/proc/modules` — strong evidence of an LKM rootkit.
|
||||
|
||||
**Contain**
|
||||
|
||||
- Assume user-space tools are compromised. Pivot to the read-only dashboard from
|
||||
**another machine** to cross-check, and prefer offline analysis.
|
||||
- Do not `rmmod` blindly — capture the module and system state first.
|
||||
|
||||
**Recover**
|
||||
|
||||
- A confirmed kernel rootkit means **rebuild from known-good media**; you cannot
|
||||
reliably clean a host whose kernel is hooked. Preserve the disk image first.
|
||||
|
||||
**False positives:** PID exit races (Sentinel re-verifies hidden PIDs to reduce
|
||||
these) and a legitimately promiscuous interface (a sniffer/monitoring tool you
|
||||
run). Confirm the owning process before escalating.
|
||||
|
||||
---
|
||||
|
||||
## Runbook 5 — Sensor tampering
|
||||
|
||||
**Triggers:** `fim_modified` on a Sentinel-owned path (self-integrity), a stale or
|
||||
missing heartbeat surfaced by the off-box `watchdog`, or the dashboard going
|
||||
unreachable.
|
||||
|
||||
Disabling the alarm is step zero of a competent intrusion — so a silent or
|
||||
edited Sentinel is itself an alert.
|
||||
|
||||
**Confirm**
|
||||
|
||||
- From the **watchdog host** (never the protected host):
|
||||
|
||||
```bash
|
||||
enodia-sentinel watchdog --url http://<host>:8787 --token <token> --max-age 120
|
||||
```
|
||||
|
||||
Exit 1 means down, unreachable, or stale.
|
||||
- On the host (if you still trust it): `enodia-sentinel status` and check whether
|
||||
Sentinel's own binaries/config/units/hook changed:
|
||||
|
||||
```bash
|
||||
enodia-sentinel fim-check # SELF_PATHS are always watched
|
||||
```
|
||||
|
||||
**Contain**
|
||||
|
||||
- If Sentinel's files were edited, treat the host as compromised: an attacker with
|
||||
the privilege to edit the sensor had the privilege to do far more.
|
||||
- Move to evidence-preservation and offline analysis rather than trusting any
|
||||
further local output.
|
||||
|
||||
**Recover**
|
||||
|
||||
- Reinstall Sentinel from a trusted source and re-baseline on a clean host.
|
||||
- Harden against recurrence: `chattr +i` on Sentinel's binaries, config, units,
|
||||
baselines, and the pacman hook; keep the watchdog running off-box.
|
||||
|
||||
**False positives:** a legitimate Sentinel upgrade changes its own files — but
|
||||
that should coincide with a package transaction. An edit outside one is the
|
||||
suspicious case.
|
||||
|
||||
---
|
||||
|
||||
## After any incident
|
||||
|
||||
- Keep the evidence bundle; do not delete snapshots while an investigation is open
|
||||
(retention is bounded by `max_snapshots` / `max_snapshot_age_days`).
|
||||
- Re-baseline (`baseline`, `fim-update`) only once the host is confirmed clean —
|
||||
baselining a compromised host blesses the compromise.
|
||||
- Re-run `enodia-sentinel posture check` to close the hygiene gaps that enabled
|
||||
the intrusion.
|
||||
- The incident-grouping commands (`incident show/export`) on the
|
||||
[roadmap](ROADMAP.md) will eventually automate the evidence-bundle steps above.
|
||||
Loading…
Add table
Add a link
Reference in a new issue