enodia-sentinal/docs/OPERATIONS.md

Enodia Sentinel Operations Guide

This guide turns the current toolset into repeatable operator workflows. It is deliberately practical: what to run, what to expect, and what to check next.

First Install Checklist

1. Install and enable the service.

   sudo make install
   sudo make enable
   

2. Build the listener and SUID baselines.

   sudo enodia-sentinel baseline
   

3. Build the FIM baseline.

   sudo enodia-sentinel fim-baseline
   

4. Run a one-shot check.

   sudo enodia-sentinel check
   sudo enodia-sentinel rootcheck
   sudo enodia-sentinel pkgdb-check
   

5. Start the dashboard if desired.

   sudo systemctl enable --now enodia-sentinel-web
   

6. Configure one off-box notification or watchdog path. A local-only alerting path is not enough if the whole host goes silent.

Routine Health Checks

Run these after upgrades, config edits, or suspicious behavior:

enodia-sentinel check
enodia-sentinel fim-check
enodia-sentinel pkgdb-check
enodia-sentinel pkgdb-verify --sample 100
enodia-sentinel rootcheck
enodia-sentinel posture check
enodia-sentinel triage

Expected healthy output:

• check: no alerts, or only expected tuned findings.
• fim-check: no changes against baseline after legitimate updates have been acknowledged.
• pkgdb-check: package DB consistent with anchor.
• pkgdb-verify: sampled files match signed cache packages.
• rootcheck: no hidden processes, modules, ports, or sniffers.
• posture check: no SSH/sudo/PATH/permission/signature hygiene findings, or only ones you have consciously accepted (e.g. password auth on a host that needs it).

Alert Workflow

When Sentinel fires:

1. Open the newest JSON and text snapshots in /var/log/enodia-sentinel.

2. Identify the signature, sid, affected PIDs, executable paths, remote peers, and parent process.

3. Run enodia-sentinel triage to separate known benign listener noise from findings that need review.

4. If the alert involves a binary, run:

   enodia-sentinel fim-check --packages
   enodia-sentinel pkgdb-verify --sample 200
   

5. If the alert involves hiding or tampering, run:

   enodia-sentinel rootcheck
   enodia-sentinel pkgdb-check
   

6. Preserve evidence before changing state:

   cp -a /var/log/enodia-sentinel /tmp/enodia-sentinel-evidence
   

7. Contain manually for now. Future response commands should produce a dry-run plan first.

Common Findings

Finding	First checks
reverse_shell	Inspect parent process, fd table, remote peer, shell argv, and user.
egress	Confirm interpreter, destination IP, command line, and parent process.
new_listener	Identify owning process and whether the binary is package-owned.
new_suid	Check path, owner, package provenance, and whether it lives in a writable directory.
persistence	Diff the changed file and identify the modifying package or process if possible.
fim_modified	Verify package ownership, package checksum, and whether the change followed an upgrade.
pkgdb_tamper	Treat as high-confidence tampering until a legitimate package transaction explains it.
pkg_signature_mismatch	Treat as a potentially trojaned package-owned file.
hidden_* rootcheck alerts	Compare live tools, preserve evidence, and consider offline analysis.

Baseline Hygiene

Baselines are useful only if they represent a known-good state.

• Build baselines after installing Sentinel on a clean host.
• Rebuild listener/SUID baselines after intentionally adding long-running services or privileged binaries.
• Use fim-update only after confirming changes are legitimate. Package hooks already run it for normal package transactions.
• Keep a copy of important anchors off-box when possible.

Suggested Hardening

• Keep the dashboard bound to Tailscale or loopback, not a public interface.
• Set web_token explicitly for fully read-only service operation.
• Enable at least one push backend.
• Run watchdog from another machine.
• Consider chattr +i for Sentinel binaries, systemd units, config, baselines, and the pacman hook after setup.
• Keep the package cache populated if using signed-package verification.
• Do not disable package signature verification.

Evidence Export

Until the incident command set exists, an evidence bundle is simply:

tar -C /var/log -czf /tmp/enodia-sentinel-evidence.tgz enodia-sentinel

That bundle contains events, snapshots, baselines, heartbeat state, package DB anchor, and dashboard-readable JSON.