enodia-sentinal/docs/OPERATIONS.md

Enodia Sentinel Operations Guide

This guide turns the current toolset into repeatable operator workflows. It is deliberately practical: what to run, what to expect, and what to check next.

First Install Checklist

1. Install and enable the service.

   sudo make install
   sudo make enable
   

2. Build the listener and SUID baselines.

   sudo enodia-sentinel baseline
   

3. Build the FIM baseline.

   sudo enodia-sentinel fim-baseline
   

4. Run a one-shot check.

   sudo enodia-sentinel check
   sudo enodia-sentinel rootcheck
   sudo enodia-sentinel pkgdb-check
   

5. Start the dashboard if desired.

   sudo systemctl enable --now enodia-sentinel-web
   

   Open the https://... URL from the service log. The default certificate is self-signed, so add a browser exception or configure web_tls_cert and web_tls_key with your own local/private CA certificate.

6. Configure one off-box notification or watchdog path. A local-only alerting path is not enough if the whole host goes silent.

Routine Health Checks

Run these after upgrades, config edits, or suspicious behavior:

enodia-sentinel check
enodia-sentinel fim-check
enodia-sentinel pkgdb-check
enodia-sentinel pkgdb-verify --sample 100
enodia-sentinel rootcheck
enodia-sentinel posture check
enodia-sentinel triage

Expected healthy output:

• check: no alerts, or only expected tuned findings.
• fim-check: no changes against baseline after legitimate updates have been acknowledged.
• pkgdb-check: package DB consistent with anchor.
• pkgdb-verify: sampled files match signed cache packages.
• rootcheck: no hidden processes, processes hidden from ps, hidden modules/ports/raw/special-protocol sockets, sniffers, known rootkit modules, raw ICMP/SCTP-style channels, or unexplained kernel/module taint.
• posture check: no SSH/sudo/PATH/permission/signature hygiene findings, or only ones you have consciously accepted (e.g. password auth on a host that needs it).

The HTTPS dashboard shows the same posture findings under the Posture tab and summarizes FIM/package anchors plus heartbeat freshness under the Integrity tab for quick remote review.

Alert Workflow

When Sentinel fires:

1. Open the newest JSON and text snapshots in /var/log/enodia-sentinel.

2. Identify the signature, sid, affected PIDs, executable paths, remote peers, and parent process.

3. Run enodia-sentinel triage to separate known benign listener noise from findings that need review.

4. Build a read-only response plan from the grouped incident:

   enodia-sentinel incident list
   enodia-sentinel respond plan <incident-id>
   

   Review the plan before acting; Sentinel does not execute containment commands. CLI-generated plans are saved under /var/log/enodia-sentinel/response-plans/ and indexed in /var/log/enodia-sentinel/response-audit.log for later review.

5. If the alert involves a binary, run:

   enodia-sentinel fim-check --packages
   enodia-sentinel pkgdb-verify --sample 200
   

6. If the alert involves hiding or tampering, run:

   enodia-sentinel rootcheck
   enodia-sentinel pkgdb-check
   

7. Preserve evidence before changing state:

   cp -a /var/log/enodia-sentinel /tmp/enodia-sentinel-evidence
   

8. Contain manually for now, following the matching playbook in RUNBOOKS.md. The response plan gives reviewed commands, but applying them remains an explicit operator action.

Common Findings

Finding	First checks
reverse_shell	Inspect parent process, fd table, remote peer, shell argv, and user.
egress	Confirm interpreter, destination IP, command line, and parent process.
new_listener	Identify owning process and whether the binary is package-owned.
new_suid	Check path, owner, package provenance, and whether it lives in a writable directory.
persistence	Diff the changed file and identify the modifying package or process if possible.
fim_modified	Verify package ownership, package checksum, and whether the change followed an upgrade.
pkgdb_tamper	Treat as high-confidence tampering until a legitimate package transaction explains it.
pkg_signature_mismatch	Treat as a potentially trojaned package-owned file.
rootkit_* / kernel_tainted alerts	Compare live tools, review module provenance, preserve evidence, and consider offline analysis.

Baseline Hygiene

Baselines are useful only if they represent a known-good state.

• Build baselines after installing Sentinel on a clean host.
• Rebuild listener/SUID baselines after intentionally adding long-running services or privileged binaries.
• Use fim-update only after confirming changes are legitimate. Package hooks already run it for normal package transactions.
• Keep a copy of important anchors off-box when possible.
• Prefer baseline accept <kind> <target> --reason "..." over rebuilding a whole baseline when only one item changed legitimately (an edited config, a new service port, an installed SUID helper). Unlike a rebuild, the acceptance is per-item, carries a recorded reason, is fingerprint-bound (FIM/package items re-alert if the file changes again), and can be time-boxed with --expires. Review outstanding acceptances with baseline list; it exits non-zero and marks rows stale when an accepted item has drifted again or its TTL lapsed.

Suggested Hardening

• Keep the dashboard bound to Tailscale or loopback, not a public interface.
• Set web_token explicitly for fully read-only service operation.
• Keep HTTPS enabled. The built-in self-signed certificate is acceptable for a single host after you add an exception; use a private/local CA for cleaner browser trust.
• Enable at least one push backend.
• Run watchdog from another machine.
• Consider chattr +i for Sentinel binaries, systemd units, config, baselines, and the pacman hook after setup.
• Keep the package cache populated if using signed-package verification.
• Do not disable package signature verification.

Evidence Export

Until the incident command set exists, an evidence bundle is simply:

tar -C /var/log -czf /tmp/enodia-sentinel-evidence.tgz enodia-sentinel

That bundle contains events, snapshots, baselines, heartbeat state, package DB anchor, and dashboard-readable JSON.