luna/enodia-sentinal
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
enodia-sentinal/docs/RUNBOOKS.md

9 KiB
Raw Blame History

Enodia Sentinel Incident Response Runbooks

Practical, conservative response playbooks for the alerts Sentinel raises. Each runbook follows the same arc: confirm → preserve → contain → recover, and prefers reversible, manual steps. Sentinel does not perform automatic containment (see THREAT_MODEL.md); these are operator actions.

A note on trust: if you suspect kernel-level compromise, remember that the tools you run (ss, ps, even ls) may be lying to you. Cross-check from the dashboard on another machine, and when in doubt, capture evidence and analyze the disk offline.

The First Five Minutes

For any alert, before changing anything:

  1. Read the snapshot. Every alert writes alert-*.log (human) and alert-*.json (machine) under /var/log/enodia-sentinel. Note the signature, sid, severity, PIDs, executable paths, and remote peers.

  2. Get host state. enodia-sentinel status — is the daemon healthy, how many alerts, when was the last.

  3. Triage known noise. enodia-sentinel triage separates likely false positives from findings that need a human.

  4. Build a dry-run response plan. Find the incident and review the proposed actions:

    enodia-sentinel incident list
enodia-sentinel respond plan <incident-id>

  5. Preserve evidence before you touch anything:

    tar -C /var/log -czf /tmp/enodia-evidence-$(date +%s).tgz enodia-sentinel

Then pick the runbook below that matches the signature.

Runbook 1 — Reverse shell / suspicious egress

Triggers: reverse_shell (sid 100010), egress (sid 100016).

An interpreter wired to a socket, or an interpreter holding a connection to a public IP — the classic interactive-C2 and beaconing shapes.

Confirm

  • Open the snapshot; identify the PID, comm, parent process, and remote peer.

  • Inspect live (these views may be untrustworthy if root is compromised):

    ps -o pid,ppid,user,comm,args -p <pid>
ls -l /proc/<pid>/fd        # which fds are the socket
cat /proc/<pid>/environ | tr '\0' '\n'
ss -tnp | grep <pid>        # the peer

  • Is the peer expected? Compare against egress_allow_cidrs in config. A package manager or browser to a public IP is normal; bash/python/nc is not.

Contain

  • Capture the process before killing it (memory/maps if you have tooling), then: kill -STOP <pid> to freeze, inspect, and only then kill -9.
  • Block the peer if it is hostile: nft add rule inet filter output ip daddr <peer> drop.
  • Find persistence the shell may have dropped — run Runbook 2.

Recover

  • Rotate any credentials that were reachable from the compromised process.
  • Rebuild listener/SUID baselines only after the host is confirmed clean.

False positives: legitimate admin scripts, monitoring agents, or P2P clients. Add trusted egress destinations to egress_allow_cidrs rather than disabling the detector.

Runbook 2 — Persistence tampering

Triggers: persistence (sid 100015), fim_modified on a watched persistence path.

A change to cron, a systemd unit, an SSH authorized_keys, a shell rc file, or sudo/account config — how an attacker survives a reboot.

Confirm

  • The snapshot names the changed path. Diff it against the FIM baseline and package ownership:

    enodia-sentinel fim-check
enodia-sentinel fim-check --packages     # does a package own & vouch for it?

  • For SSH keys: is the added key yours? For systemd/cron: does the unit run a binary in a writable or unusual location?

Contain

  • Remove the unauthorized entry (the added key, cron line, or unit). Keep a copy in your evidence bundle first.
  • systemctl disable --now <unit> for a malicious service; check what it spawned.

Recover

  • After removing persistence, hunt for the process it launched (Runbook 1) and any trojaned binaries (Runbook 3).
  • Re-anchor the baseline only once clean: enodia-sentinel fim-update.

False positives: legitimate package upgrades and admin edits. The pacman hook auto-runs fim-update for normal transactions; an unexplained change outside a transaction is the suspicious case.

Runbook 3 — Trojaned binary / package tampering

Triggers: fim_modified / fim_pkg_modified on a system binary, pkgdb_tamper (package DB edited out of band), pkg_signature_mismatch (on-disk file ≠ signed package), pacman_siglevel_disabled.

Confirm

  • Layer the integrity checks from weakest to strongest trust anchor:

    enodia-sentinel fim-check --packages   # vs FIM baseline + pacman -Qkk
enodia-sentinel pkgdb-check            # was the local checksum DB rewritten?
enodia-sentinel pkgdb-verify --sample 200   # vs signed cache packages (.MTREE)

  • pkg_signature_mismatch is the strongest signal: it bypasses the mutable local DB. Treat a mismatch on a system binary as a trojaned file until proven otherwise. pkgdb_tamper suggests someone tried to hide exactly this.

Contain

  • Do not execute the suspect binary. If it is currently running, freeze it (Runbook 1).
  • Confirm package-signature policy is intact: enodia-sentinel posture check (flags pacman_siglevel_disabled).

Recover

  • Identify the owning package and reinstall from a trusted source:

    pacman -Qo /path/to/binary
pacman -S --overwrite '*' <package>     # reinstall the verified file
enodia-sentinel fim-check --packages    # confirm it now matches

  • Then re-anchor: enodia-sentinel fim-update and re-run pkgdb-check.

False positives: a mid-upgrade snapshot can momentarily disagree. Re-run after the transaction completes; a persistent mismatch is the real finding.

Runbook 4 — Hidden listener / rootkit

Triggers: rootkit_hidden_process (100022), rootkit_hidden_module (100023), rootkit_hidden_port (100024), promiscuous_interface (100025), and new_listener (100013) for an unexplained open port.

These come from cross-view checks: the same question asked two ways, answered differently, because something is hiding.

Confirm

  • Re-run the cross-view sweep and capture it:

    enodia-sentinel rootcheck

  • new_listener: identify the owning process and whether its binary is package-owned. A hidden port that rootcheck sees but ss does not is far more serious than a forgotten service.

  • A rootkit_hidden_module names a live kernel module absent from /proc/modules — strong evidence of an LKM rootkit.

Contain

  • Assume user-space tools are compromised. Pivot to the read-only dashboard from another machine to cross-check, and prefer offline analysis.
  • Do not rmmod blindly — capture the module and system state first.

Recover

  • A confirmed kernel rootkit means rebuild from known-good media; you cannot reliably clean a host whose kernel is hooked. Preserve the disk image first.

False positives: PID exit races (Sentinel re-verifies hidden PIDs to reduce these) and a legitimately promiscuous interface (a sniffer/monitoring tool you run). Confirm the owning process before escalating.

Runbook 5 — Sensor tampering

Triggers: fim_modified on a Sentinel-owned path (self-integrity), a stale or missing heartbeat surfaced by the off-box watchdog, or the dashboard going unreachable.

Disabling the alarm is step zero of a competent intrusion — so a silent or edited Sentinel is itself an alert.

Confirm

  • From the watchdog host (never the protected host):

    enodia-sentinel watchdog --url https://<host>:8787 --token <token> --max-age 120

    Exit 1 means down, unreachable, or stale.

  • On the host (if you still trust it): enodia-sentinel status and check whether Sentinel's own binaries/config/units/hook changed:

    enodia-sentinel fim-check        # SELF_PATHS are always watched

Contain

  • If Sentinel's files were edited, treat the host as compromised: an attacker with the privilege to edit the sensor had the privilege to do far more.
  • Move to evidence-preservation and offline analysis rather than trusting any further local output.

Recover

  • Reinstall Sentinel from a trusted source and re-baseline on a clean host.
  • Harden against recurrence: chattr +i on Sentinel's binaries, config, units, baselines, and the pacman hook; keep the watchdog running off-box.

False positives: a legitimate Sentinel upgrade changes its own files — but that should coincide with a package transaction. An edit outside one is the suspicious case.

After any incident

  • Keep the evidence bundle; do not delete snapshots while an investigation is open (retention is bounded by max_snapshots / max_snapshot_age_days).
  • Re-baseline (baseline, fim-update) only once the host is confirmed clean — baselining a compromised host blesses the compromise.
  • Re-run enodia-sentinel posture check to close the hygiene gaps that enabled the intrusion.
  • The incident-grouping commands (incident show/export) on the roadmap will eventually automate the evidence-bundle steps above.