luna/enodia-sentinal
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
enodia-sentinal/docs/RUNBOOKS.md

16 KiB
Raw Blame History

Enodia Sentinel Incident Response Runbooks

Practical, conservative response playbooks for the alerts Sentinel raises. Each runbook follows the same arc: confirm → preserve → contain → recover, and prefers reversible, manual steps. Sentinel does not perform automatic containment (see THREAT_MODEL.md); these are operator actions.

A note on trust: if you suspect kernel-level compromise, remember that the tools you run (ss, ps, even ls) may be lying to you. Cross-check from the dashboard on another machine, and when in doubt, capture evidence and analyze the disk offline.

The First Five Minutes

For any alert, before changing anything:

  1. Read the snapshot. Every alert writes alert-*.log (human) and alert-*.json (machine) under /var/log/enodia-sentinel. Note the signature, sid, severity, PIDs, executable paths, and remote peers.

  2. Get host state. enodia-sentinel status — is the daemon healthy, how many alerts, when was the last.

  3. Triage known noise. enodia-sentinel triage separates likely false positives from findings that need a human.

  4. Build a dry-run response plan. Find the incident and review the proposed actions:

    enodia-sentinel incident list
enodia-sentinel respond plan <incident-id>

    The CLI writes a copy under /var/log/enodia-sentinel/response-plans/ and appends /var/log/enodia-sentinel/response-audit.log, so the reviewed plan can be handed off or compared with later actions.

  5. Preserve evidence before you touch anything:

    tar -C /var/log -czf /tmp/enodia-evidence-$(date +%s).tgz enodia-sentinel

Then pick the runbook below that matches the signature.

Correlation Alerts

Triggers: correlation.multi_stage_intrusion (sid 100080).

Correlation alerts are not separate raw telemetry. They mean one incident now contains multiple lower-level signals that form a stronger story, such as a web/database service spawning an interpreter and the same incident showing suspicious egress or an unusual listener inside the correlation window.

  • Treat the correlated incident severity as the investigation priority.

  • Keep reviewing the underlying raw alerts; they remain in the incident timeline and snapshots.

  • Export the incident before containment:

    enodia-sentinel incident export <incident-id> > /tmp/<incident-id>.json

  • If the correlation involves a web/database parent, preserve service logs, access logs, and application deployment state before restarting the service.

Runbook 1 — Reverse shell / suspicious egress

Triggers: reverse_shell (sid 100010), egress (sid 100016), stealth_network (sid 100036).

An interpreter wired to a socket, an interpreter holding a connection to a public IP, or activity over raw/SCTP/DCCP/packet/XDP/TIPC/vsock families - the classic interactive-C2, beaconing, and covert-channel shapes.

Confirm

  • Open the snapshot; identify the PID, comm, parent process, and remote peer.

  • Inspect live (these views may be untrustworthy if root is compromised):

    ps -o pid,ppid,user,comm,args -p <pid>
ls -l /proc/<pid>/fd        # which fds are the socket
cat /proc/<pid>/environ | tr '\0' '\n'
ss -tnp | grep <pid>        # the peer

  • Is the peer expected? Compare against egress_allow_cidrs in config. A package manager or browser to a public IP is normal; bash/python/nc is not.

  • For stealth_network, check the socket family and owner. Packet/raw sockets are normal for approved sniffers and some network managers; SCTP/DCCP/TIPC/XDP on a desktop or ordinary server should have a clear service owner. Tune with stealth_network_allow_comms or, for intentionally used families, stealth_network_allow_kinds.

Contain

  • Capture the process before killing it (memory/maps if you have tooling), then: kill -STOP <pid> to freeze, inspect, and only then kill -9.
  • Block the peer if it is hostile: nft add rule inet filter output ip daddr <peer> drop.
  • Find persistence the shell may have dropped — run Runbook 2.

Recover

  • Rotate any credentials that were reachable from the compromised process.
  • Rebuild listener/SUID baselines only after the host is confirmed clean.

False positives: legitimate admin scripts, monitoring agents, or P2P clients. Add trusted egress destinations to egress_allow_cidrs rather than disabling the detector.

Runbook 2 — Persistence tampering

Triggers: persistence (sid 100015), fim_modified on a watched persistence path, systemd_unit_writable (sid 100050), systemd_exec_unsafe_path (sid 100051).

A change to cron, a systemd unit, an SSH authorized_keys, a shell rc file, or sudo/account config — how an attacker survives a reboot.

Confirm

  • The snapshot names the changed path. Diff it against the FIM baseline and package ownership:

    enodia-sentinel fim-check
enodia-sentinel fim-check --packages     # does a package own & vouch for it?

  • For SSH keys: is the added key yours? For systemd/cron: does the unit run a binary in a writable or unusual location?

Contain

  • Remove the unauthorized entry (the added key, cron line, or unit). Keep a copy in your evidence bundle first.
  • systemctl disable --now <unit> for a malicious service; check what it spawned.

Recover

  • After removing persistence, hunt for the process it launched (Runbook 1) and any trojaned binaries (Runbook 3).
  • Re-anchor the baseline only once clean: enodia-sentinel fim-update.

False positives: legitimate package upgrades and admin edits. The pacman hook auto-runs fim-update for normal transactions; an unexplained change outside a transaction is the suspicious case.

Runbook 3 — Trojaned binary / package tampering

Triggers: fim_modified / fim_pkg_modified on a system binary, pkgdb_tamper (package DB edited out of band), pkg_signature_mismatch (on-disk file ≠ signed package), pacman_siglevel_disabled.

Confirm

  • Layer the integrity checks from weakest to strongest trust anchor:

    enodia-sentinel fim-check --packages   # vs FIM baseline + pacman -Qkk
enodia-sentinel pkgdb-check            # was the local checksum DB rewritten?
enodia-sentinel pkgdb-verify --sample 200   # vs signed cache packages (.MTREE)

  • pkg_signature_mismatch is the strongest signal: it bypasses the mutable local DB. Treat a mismatch on a system binary as a trojaned file until proven otherwise. pkgdb_tamper suggests someone tried to hide exactly this.

Contain

  • Do not execute the suspect binary. If it is currently running, freeze it (Runbook 1).
  • Confirm package-signature policy is intact: enodia-sentinel posture check (flags pacman_siglevel_disabled).

Recover

  • Identify the owning package and reinstall from a trusted source:

    enodia-sentinel respond plan <incident-id>
pacman -Qo /path/to/binary
enodia-sentinel pkgdb-verify
pacman -S --overwrite '*' <package>     # reinstall the verified file
enodia-sentinel fim-check --packages    # confirm it now matches
enodia-sentinel fim-update              # re-anchor only after review

  • Then re-run pkgdb-check.

False positives: a mid-upgrade snapshot can momentarily disagree. Re-run after the transaction completes; a persistent mismatch is the real finding.

Runbook 4 — Hidden listener / rootkit

Triggers: rootkit_hidden_process (100022), rootkit_hidden_module (100023), rootkit_hidden_port (100024), promiscuous_interface (100025), rootkit_known_module (100028), rootkit_tainted_module (100029), kernel_tainted (100030), rootkit_hidden_udp_port (100031), rootkit_hidden_raw_socket (100034), raw_icmp_socket (100035), rootkit_hidden_protocol_socket (100037), rootkit_ps_hidden_process (100038), and new_listener (100013) for an unexplained open port.

These come from cross-view checks: the same question asked two ways, answered differently, because something is hiding.

Confirm

  • Re-run the cross-view sweep and capture it:

    enodia-sentinel rootcheck

  • new_listener: identify the owning process and whether its binary is package-owned. A hidden port that rootcheck sees but ss does not is far more serious than a forgotten service.

  • rootkit_ps_hidden_process: compare /proc/<pid>/cmdline, /proc/<pid>/exe, and /proc/<pid>/maps against ps/pgrep output. A process visible in /proc but missing from ps strongly suggests userland process-tool hiding.

  • rootkit_hidden_udp_port: check whether a legitimate UDP service owns the port; a port present in /proc/net/udp* but absent from ss -u suggests tool output may be hooked.

  • rootkit_hidden_raw_socket or raw_icmp_socket: look for authorized ping, monitoring, traceroute, or packet tooling first. A persistent raw ICMP socket with no operator explanation is consistent with ICMP knockers/backdoors and should be investigated with process fd tables, package provenance, and memory capture before killing the process.

  • rootkit_hidden_protocol_socket: inspect SCTP, DCCP, packet, TIPC, or XDP use. If /proc/net sees the family but ss does not, treat normal socket tooling as suspect and pivot to offline evidence or the dashboard from another host.

  • A rootkit_hidden_module names a live kernel module absent from /proc/modules — strong evidence of an LKM rootkit.

  • A rootkit_known_module finding names a module associated with public LKM rootkits; treat it as a high-confidence compromise until proven otherwise.

  • A rootkit_tainted_module or kernel_tainted finding may be legitimate vendor/DKMS software, but unsigned/out-of-tree or force-loaded modules should be explained and allowlisted only after review.

Contain

  • Assume user-space tools are compromised. Pivot to the read-only dashboard from another machine to cross-check, and prefer offline analysis.
  • Do not rmmod blindly — capture the module and system state first.

Recover

  • A confirmed kernel rootkit means rebuild from known-good media; you cannot reliably clean a host whose kernel is hooked. Preserve the disk image first.

False positives: PID exit races (Sentinel re-verifies hidden PIDs to reduce these), a legitimately promiscuous interface (a sniffer/monitoring tool you run), and expected tainted vendor/DKMS modules. Confirm the owning process or module provenance before escalating; use rootcheck_module_allow only for modules you intentionally trust.

Runbook 5 — Memory obfuscation, process hiding, or injection library

Triggers: memory_obfuscation (100039), process_hiding_library (100048), process_injection_library (100049)

These alerts come from /proc/<pid>/maps. They do not read process memory; they flag map shapes associated with encrypted/packed payloads or userland hiding: RWX pages, executable anonymous memory, executable memfd/deleted mappings, or libraries with process-hiding names such as libhide. They also flag executable shared-library mappings from writable/runtime paths such as /tmp, /dev/shm, /var/tmp, and /run/user, which are common places for injected userland hooks or staged .so payloads.

Confirm

  • Open the snapshot and inspect cmdline, exe, parent process, and captured memory maps for the PID.
  • For process_hiding_library, check whether the mapped library is package-owned and whether it is loaded into tools like ps, pgrep, ss, shells, or admin utilities.
  • For process_injection_library, preserve the mapped .so, check its owner, hash, parent process, and whether it was deliberately loaded by a known profiler, debugger, or local instrumentation tool.
  • For memory_obfuscation, distinguish expected JIT runtimes from unknown implants. Tune known runtimes with memory_obfuscation_allow_comms; use memory_obfuscation_allow_paths only for well-understood local runtimes or instrumentation.

Contain

  • Preserve the executable and /proc/<pid>/maps before killing. If the alert is for executable memfd or deleted mappings, capture memory first if you have trusted tooling; the on-disk payload may be gone.
  • Re-run rootcheck after containment to confirm no process-tool hiding or hidden sockets remain.

Runbook 6 — Credential or input capture

Triggers: credential_access (100033), input_snooper (100032)

These alerts mean a non-allowlisted process has credential material or keyboard input devices open. They map to Gonzalo-style keylogging and local credential harvesting, but are intentionally behavior-based.

Confirm

  • Open the snapshot for the alert and inspect the captured cmdline, exe, cwd, parent process, and fd target.
  • For input_snooper, confirm whether the process is an expected compositor, display server, input remapper, or accessibility tool. Add stable legitimate process names to input_snooper_allow_comms.
  • For credential_access, identify which secret was opened: shadow DB, private SSH key, browser credential store, or NetworkManager secret profile. Add only well-understood auth/keyring/browser readers to credential_access_allow_comms.

Contain

  • Preserve evidence first for unknown processes: copy the executable path if it still exists, capture /proc/<pid>/maps, and export the incident.
  • Rotate exposed credentials and SSH keys after containment; assume browser session cookies and saved passwords may be compromised if browser stores were opened by an unknown process.

Runbook 7 — Sensor tampering

Triggers: fim_modified on a Sentinel-owned path (self-integrity), a stale or missing heartbeat surfaced by the off-box watchdog, or the dashboard going unreachable.

Disabling the alarm is step zero of a competent intrusion — so a silent or edited Sentinel is itself an alert.

Confirm

  • From the watchdog host (never the protected host):

    enodia-sentinel watchdog --url https://<host>:8787 --token <token> --max-age 120

    Exit 1 means down, unreachable, or stale.

  • On the host (if you still trust it): enodia-sentinel status and check whether Sentinel's own binaries/config/units/hook changed:

    enodia-sentinel fim-check        # SELF_PATHS are always watched

Contain

  • If Sentinel's files were edited, treat the host as compromised: an attacker with the privilege to edit the sensor had the privilege to do far more.
  • Move to evidence-preservation and offline analysis rather than trusting any further local output.

Recover

  • Reinstall Sentinel from a trusted source and re-baseline on a clean host.
  • Harden against recurrence: chattr +i on Sentinel's binaries, config, units, baselines, and the pacman hook; keep the watchdog running off-box.

False positives: a legitimate Sentinel upgrade changes its own files — but that should coincide with a package transaction. An edit outside one is the suspicious case.

After any incident

  • Keep the evidence bundle; do not delete snapshots while an investigation is open (retention is bounded by max_snapshots / max_snapshot_age_days).
  • Preserve hash-chain.jsonl with the snapshots. The chain records the local digest sequence for snapshot logs, snapshot JSON, and events.log; a missing or broken chain is not proof of compromise by itself, but it is a reason to distrust local evidence and compare with off-box copies.
  • Re-baseline (baseline, fim-update) only once the host is confirmed clean — baselining a compromised host blesses the compromise.
  • Re-run enodia-sentinel posture check to close the hygiene gaps that enabled the intrusion.
  • Use enodia-sentinel incident show/export and the saved response plan as the durable handoff record for later review.