318 lines
12 KiB
Markdown
318 lines
12 KiB
Markdown
# Enodia Sentinel Incident Response Runbooks
|
|
|
|
Practical, conservative response playbooks for the alerts Sentinel raises. Each
|
|
runbook follows the same arc: **confirm → preserve → contain → recover**, and
|
|
prefers reversible, manual steps. Sentinel does not perform automatic
|
|
containment (see [THREAT_MODEL.md](THREAT_MODEL.md)); these are operator actions.
|
|
|
|
> A note on trust: if you suspect kernel-level compromise, remember that the
|
|
> tools you run (`ss`, `ps`, even `ls`) may be lying to you. Cross-check from the
|
|
> dashboard on another machine, and when in doubt, capture evidence and analyze
|
|
> the disk offline.
|
|
|
|
## The First Five Minutes
|
|
|
|
For any alert, before changing anything:
|
|
|
|
1. **Read the snapshot.** Every alert writes `alert-*.log` (human) and
|
|
`alert-*.json` (machine) under `/var/log/enodia-sentinel`. Note the
|
|
`signature`, `sid`, severity, PIDs, executable paths, and remote peers.
|
|
2. **Get host state.** `enodia-sentinel status` — is the daemon healthy, how many
|
|
alerts, when was the last.
|
|
3. **Triage known noise.** `enodia-sentinel triage` separates likely false
|
|
positives from findings that need a human.
|
|
4. **Build a dry-run response plan.** Find the incident and review the proposed
|
|
actions:
|
|
|
|
```bash
|
|
enodia-sentinel incident list
|
|
enodia-sentinel respond plan <incident-id>
|
|
```
|
|
|
|
5. **Preserve evidence before you touch anything:**
|
|
|
|
```bash
|
|
tar -C /var/log -czf /tmp/enodia-evidence-$(date +%s).tgz enodia-sentinel
|
|
```
|
|
|
|
Then pick the runbook below that matches the signature.
|
|
|
|
---
|
|
|
|
## Runbook 1 — Reverse shell / suspicious egress
|
|
|
|
**Triggers:** `reverse_shell` (sid 100010), `egress` (sid 100016),
|
|
`stealth_network` (sid 100036).
|
|
|
|
An interpreter wired to a socket, an interpreter holding a connection to a
|
|
public IP, or activity over raw/SCTP/DCCP/packet/XDP/TIPC/vsock families - the
|
|
classic interactive-C2, beaconing, and covert-channel shapes.
|
|
|
|
**Confirm**
|
|
|
|
- Open the snapshot; identify the PID, `comm`, parent process, and remote peer.
|
|
- Inspect live (these views may be untrustworthy if root is compromised):
|
|
|
|
```bash
|
|
ps -o pid,ppid,user,comm,args -p <pid>
|
|
ls -l /proc/<pid>/fd # which fds are the socket
|
|
cat /proc/<pid>/environ | tr '\0' '\n'
|
|
ss -tnp | grep <pid> # the peer
|
|
```
|
|
|
|
- Is the peer expected? Compare against `egress_allow_cidrs` in config. A package
|
|
manager or browser to a public IP is normal; `bash`/`python`/`nc` is not.
|
|
- For `stealth_network`, check the socket family and owner. Packet/raw sockets
|
|
are normal for approved sniffers and some network managers; SCTP/DCCP/TIPC/XDP
|
|
on a desktop or ordinary server should have a clear service owner. Tune with
|
|
`stealth_network_allow_comms` or, for intentionally used families,
|
|
`stealth_network_allow_kinds`.
|
|
|
|
**Contain**
|
|
|
|
- Capture the process before killing it (memory/maps if you have tooling), then:
|
|
`kill -STOP <pid>` to freeze, inspect, and only then `kill -9`.
|
|
- Block the peer if it is hostile: `nft add rule inet filter output ip daddr <peer> drop`.
|
|
- Find persistence the shell may have dropped — run Runbook 2.
|
|
|
|
**Recover**
|
|
|
|
- Rotate any credentials that were reachable from the compromised process.
|
|
- Rebuild listener/SUID baselines only after the host is confirmed clean.
|
|
|
|
**False positives:** legitimate admin scripts, monitoring agents, or P2P clients.
|
|
Add trusted egress destinations to `egress_allow_cidrs` rather than disabling the
|
|
detector.
|
|
|
|
---
|
|
|
|
## Runbook 2 — Persistence tampering
|
|
|
|
**Triggers:** `persistence` (sid 100015), `fim_modified` on a watched persistence
|
|
path.
|
|
|
|
A change to cron, a systemd unit, an SSH `authorized_keys`, a shell rc file, or
|
|
sudo/account config — how an attacker survives a reboot.
|
|
|
|
**Confirm**
|
|
|
|
- The snapshot names the changed path. Diff it against the FIM baseline and
|
|
package ownership:
|
|
|
|
```bash
|
|
enodia-sentinel fim-check
|
|
enodia-sentinel fim-check --packages # does a package own & vouch for it?
|
|
```
|
|
|
|
- For SSH keys: is the added key yours? For systemd/cron: does the unit run a
|
|
binary in a writable or unusual location?
|
|
|
|
**Contain**
|
|
|
|
- Remove the unauthorized entry (the added key, cron line, or unit). Keep a copy
|
|
in your evidence bundle first.
|
|
- `systemctl disable --now <unit>` for a malicious service; check what it spawned.
|
|
|
|
**Recover**
|
|
|
|
- After removing persistence, hunt for the process it launched (Runbook 1) and
|
|
any trojaned binaries (Runbook 3).
|
|
- Re-anchor the baseline only once clean: `enodia-sentinel fim-update`.
|
|
|
|
**False positives:** legitimate package upgrades and admin edits. The pacman hook
|
|
auto-runs `fim-update` for normal transactions; an unexplained change outside a
|
|
transaction is the suspicious case.
|
|
|
|
---
|
|
|
|
## Runbook 3 — Trojaned binary / package tampering
|
|
|
|
**Triggers:** `fim_modified` / `fim_pkg_modified` on a system binary,
|
|
`pkgdb_tamper` (package DB edited out of band), `pkg_signature_mismatch`
|
|
(on-disk file ≠ signed package), `pacman_siglevel_disabled`.
|
|
|
|
**Confirm**
|
|
|
|
- Layer the integrity checks from weakest to strongest trust anchor:
|
|
|
|
```bash
|
|
enodia-sentinel fim-check --packages # vs FIM baseline + pacman -Qkk
|
|
enodia-sentinel pkgdb-check # was the local checksum DB rewritten?
|
|
enodia-sentinel pkgdb-verify --sample 200 # vs signed cache packages (.MTREE)
|
|
```
|
|
|
|
- `pkg_signature_mismatch` is the strongest signal: it bypasses the mutable local
|
|
DB. Treat a mismatch on a system binary as a trojaned file until proven
|
|
otherwise. `pkgdb_tamper` suggests someone tried to hide exactly this.
|
|
|
|
**Contain**
|
|
|
|
- Do not execute the suspect binary. If it is currently running, freeze it
|
|
(Runbook 1).
|
|
- Confirm package-signature policy is intact: `enodia-sentinel posture check`
|
|
(flags `pacman_siglevel_disabled`).
|
|
|
|
**Recover**
|
|
|
|
- Identify the owning package and reinstall from a trusted source:
|
|
|
|
```bash
|
|
pacman -Qo /path/to/binary
|
|
pacman -S --overwrite '*' <package> # reinstall the verified file
|
|
enodia-sentinel fim-check --packages # confirm it now matches
|
|
```
|
|
|
|
- Then re-anchor: `enodia-sentinel fim-update` and re-run `pkgdb-check`.
|
|
|
|
**False positives:** a mid-upgrade snapshot can momentarily disagree. Re-run after
|
|
the transaction completes; a persistent mismatch is the real finding.
|
|
|
|
---
|
|
|
|
## Runbook 4 — Hidden listener / rootkit
|
|
|
|
**Triggers:** `rootkit_hidden_process` (100022), `rootkit_hidden_module` (100023),
|
|
`rootkit_hidden_port` (100024), `promiscuous_interface` (100025),
|
|
`rootkit_known_module` (100028), `rootkit_tainted_module` (100029),
|
|
`kernel_tainted` (100030), `rootkit_hidden_udp_port` (100031),
|
|
`rootkit_hidden_raw_socket` (100034), `raw_icmp_socket` (100035),
|
|
`rootkit_hidden_protocol_socket` (100037), and `new_listener` (100013) for an
|
|
unexplained open port.
|
|
|
|
These come from cross-view checks: the same question asked two ways, answered
|
|
differently, because something is hiding.
|
|
|
|
**Confirm**
|
|
|
|
- Re-run the cross-view sweep and capture it:
|
|
|
|
```bash
|
|
enodia-sentinel rootcheck
|
|
```
|
|
|
|
- `new_listener`: identify the owning process and whether its binary is
|
|
package-owned. A hidden port that `rootcheck` sees but `ss` does not is far more
|
|
serious than a forgotten service.
|
|
- `rootkit_hidden_udp_port`: check whether a legitimate UDP service owns the
|
|
port; a port present in `/proc/net/udp*` but absent from `ss -u` suggests tool
|
|
output may be hooked.
|
|
- `rootkit_hidden_raw_socket` or `raw_icmp_socket`: look for authorized ping,
|
|
monitoring, traceroute, or packet tooling first. A persistent raw ICMP socket
|
|
with no operator explanation is consistent with ICMP knockers/backdoors and
|
|
should be investigated with process fd tables, package provenance, and memory
|
|
capture before killing the process.
|
|
- `rootkit_hidden_protocol_socket`: inspect SCTP, DCCP, packet, TIPC, or XDP
|
|
use. If `/proc/net` sees the family but `ss` does not, treat normal socket
|
|
tooling as suspect and pivot to offline evidence or the dashboard from another
|
|
host.
|
|
- A `rootkit_hidden_module` names a live kernel module absent from
|
|
`/proc/modules` — strong evidence of an LKM rootkit.
|
|
- A `rootkit_known_module` finding names a module associated with public LKM
|
|
rootkits; treat it as a high-confidence compromise until proven otherwise.
|
|
- A `rootkit_tainted_module` or `kernel_tainted` finding may be legitimate
|
|
vendor/DKMS software, but unsigned/out-of-tree or force-loaded modules should
|
|
be explained and allowlisted only after review.
|
|
|
|
**Contain**
|
|
|
|
- Assume user-space tools are compromised. Pivot to the read-only dashboard from
|
|
**another machine** to cross-check, and prefer offline analysis.
|
|
- Do not `rmmod` blindly — capture the module and system state first.
|
|
|
|
**Recover**
|
|
|
|
- A confirmed kernel rootkit means **rebuild from known-good media**; you cannot
|
|
reliably clean a host whose kernel is hooked. Preserve the disk image first.
|
|
|
|
**False positives:** PID exit races (Sentinel re-verifies hidden PIDs to reduce
|
|
these), a legitimately promiscuous interface (a sniffer/monitoring tool you
|
|
run), and expected tainted vendor/DKMS modules. Confirm the owning process or
|
|
module provenance before escalating; use `rootcheck_module_allow` only for
|
|
modules you intentionally trust.
|
|
|
|
---
|
|
|
|
## Runbook 5 — Credential or input capture
|
|
|
|
**Triggers:** `credential_access` (100033), `input_snooper` (100032)
|
|
|
|
These alerts mean a non-allowlisted process has credential material or keyboard
|
|
input devices open. They map to Gonzalo-style keylogging and local credential
|
|
harvesting, but are intentionally behavior-based.
|
|
|
|
**Confirm**
|
|
|
|
- Open the snapshot for the alert and inspect the captured `cmdline`, `exe`,
|
|
`cwd`, parent process, and fd target.
|
|
- For `input_snooper`, confirm whether the process is an expected compositor,
|
|
display server, input remapper, or accessibility tool. Add stable legitimate
|
|
process names to `input_snooper_allow_comms`.
|
|
- For `credential_access`, identify which secret was opened: shadow DB, private
|
|
SSH key, browser credential store, or NetworkManager secret profile. Add
|
|
only well-understood auth/keyring/browser readers to
|
|
`credential_access_allow_comms`.
|
|
|
|
**Contain**
|
|
|
|
- Preserve evidence first for unknown processes: copy the executable path if it
|
|
still exists, capture `/proc/<pid>/maps`, and export the incident.
|
|
- Rotate exposed credentials and SSH keys after containment; assume browser
|
|
session cookies and saved passwords may be compromised if browser stores were
|
|
opened by an unknown process.
|
|
|
|
---
|
|
|
|
## Runbook 6 — Sensor tampering
|
|
|
|
**Triggers:** `fim_modified` on a Sentinel-owned path (self-integrity), a stale or
|
|
missing heartbeat surfaced by the off-box `watchdog`, or the dashboard going
|
|
unreachable.
|
|
|
|
Disabling the alarm is step zero of a competent intrusion — so a silent or
|
|
edited Sentinel is itself an alert.
|
|
|
|
**Confirm**
|
|
|
|
- From the **watchdog host** (never the protected host):
|
|
|
|
```bash
|
|
enodia-sentinel watchdog --url https://<host>:8787 --token <token> --max-age 120
|
|
```
|
|
|
|
Exit 1 means down, unreachable, or stale.
|
|
- On the host (if you still trust it): `enodia-sentinel status` and check whether
|
|
Sentinel's own binaries/config/units/hook changed:
|
|
|
|
```bash
|
|
enodia-sentinel fim-check # SELF_PATHS are always watched
|
|
```
|
|
|
|
**Contain**
|
|
|
|
- If Sentinel's files were edited, treat the host as compromised: an attacker with
|
|
the privilege to edit the sensor had the privilege to do far more.
|
|
- Move to evidence-preservation and offline analysis rather than trusting any
|
|
further local output.
|
|
|
|
**Recover**
|
|
|
|
- Reinstall Sentinel from a trusted source and re-baseline on a clean host.
|
|
- Harden against recurrence: `chattr +i` on Sentinel's binaries, config, units,
|
|
baselines, and the pacman hook; keep the watchdog running off-box.
|
|
|
|
**False positives:** a legitimate Sentinel upgrade changes its own files — but
|
|
that should coincide with a package transaction. An edit outside one is the
|
|
suspicious case.
|
|
|
|
---
|
|
|
|
## After any incident
|
|
|
|
- Keep the evidence bundle; do not delete snapshots while an investigation is open
|
|
(retention is bounded by `max_snapshots` / `max_snapshot_age_days`).
|
|
- Re-baseline (`baseline`, `fim-update`) only once the host is confirmed clean —
|
|
baselining a compromised host blesses the compromise.
|
|
- Re-run `enodia-sentinel posture check` to close the hygiene gaps that enabled
|
|
the intrusion.
|
|
- The incident-grouping commands (`incident show/export`) on the
|
|
[roadmap](ROADMAP.md) will eventually automate the evidence-bundle steps above.
|