4015ec872b
Red-team harness: every detector now has a named drill with an expected signature/sid, shown by `sentinel-redteam --list`. Add the missing `egress` drill (the one detector without one) — explicit-only, since it is the sole drill that emits real outbound traffic. After firing, the harness runs `enodia-sentinel check` and reports PASS/MISS per poll detection, turning it into a shared regression check. Add docs/RUNBOOKS.md: confirm/preserve/contain/recover playbooks for reverse shells & egress, persistence, trojaned binaries / package tampering, hidden listeners & rootkits, and sensor tampering — each tied to the real signatures, sids, and commands. Linked from README and the OPERATIONS alert workflow. Both close the remaining smaller v0.8 roadmap items. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
148 lines
4.6 KiB
Markdown
148 lines
4.6 KiB
Markdown
# Enodia Sentinel Operations Guide
|
|
|
|
This guide turns the current toolset into repeatable operator workflows. It is
|
|
deliberately practical: what to run, what to expect, and what to check next.
|
|
|
|
## First Install Checklist
|
|
|
|
1. Install and enable the service.
|
|
|
|
```bash
|
|
sudo make install
|
|
sudo make enable
|
|
```
|
|
|
|
2. Build the listener and SUID baselines.
|
|
|
|
```bash
|
|
sudo enodia-sentinel baseline
|
|
```
|
|
|
|
3. Build the FIM baseline.
|
|
|
|
```bash
|
|
sudo enodia-sentinel fim-baseline
|
|
```
|
|
|
|
4. Run a one-shot check.
|
|
|
|
```bash
|
|
sudo enodia-sentinel check
|
|
sudo enodia-sentinel rootcheck
|
|
sudo enodia-sentinel pkgdb-check
|
|
```
|
|
|
|
5. Start the dashboard if desired.
|
|
|
|
```bash
|
|
sudo systemctl enable --now enodia-sentinel-web
|
|
```
|
|
|
|
6. Configure one off-box notification or watchdog path. A local-only alerting
|
|
path is not enough if the whole host goes silent.
|
|
|
|
## Routine Health Checks
|
|
|
|
Run these after upgrades, config edits, or suspicious behavior:
|
|
|
|
```bash
|
|
enodia-sentinel check
|
|
enodia-sentinel fim-check
|
|
enodia-sentinel pkgdb-check
|
|
enodia-sentinel pkgdb-verify --sample 100
|
|
enodia-sentinel rootcheck
|
|
enodia-sentinel posture check
|
|
enodia-sentinel triage
|
|
```
|
|
|
|
Expected healthy output:
|
|
|
|
- `check`: no alerts, or only expected tuned findings.
|
|
- `fim-check`: no changes against baseline after legitimate updates have been
|
|
acknowledged.
|
|
- `pkgdb-check`: package DB consistent with anchor.
|
|
- `pkgdb-verify`: sampled files match signed cache packages.
|
|
- `rootcheck`: no hidden processes, modules, ports, or sniffers.
|
|
- `posture check`: no SSH/sudo/PATH/permission/signature hygiene findings, or
|
|
only ones you have consciously accepted (e.g. password auth on a host that
|
|
needs it).
|
|
|
|
## Alert Workflow
|
|
|
|
When Sentinel fires:
|
|
|
|
1. Open the newest JSON and text snapshots in `/var/log/enodia-sentinel`.
|
|
2. Identify the signature, `sid`, affected PIDs, executable paths, remote peers,
|
|
and parent process.
|
|
3. Run `enodia-sentinel triage` to separate known benign listener noise from
|
|
findings that need review.
|
|
4. If the alert involves a binary, run:
|
|
|
|
```bash
|
|
enodia-sentinel fim-check --packages
|
|
enodia-sentinel pkgdb-verify --sample 200
|
|
```
|
|
|
|
5. If the alert involves hiding or tampering, run:
|
|
|
|
```bash
|
|
enodia-sentinel rootcheck
|
|
enodia-sentinel pkgdb-check
|
|
```
|
|
|
|
6. Preserve evidence before changing state:
|
|
|
|
```bash
|
|
cp -a /var/log/enodia-sentinel /tmp/enodia-sentinel-evidence
|
|
```
|
|
|
|
7. Contain manually for now, following the matching playbook in
|
|
[RUNBOOKS.md](RUNBOOKS.md). Future response commands should produce a dry-run
|
|
plan first.
|
|
|
|
## Common Findings
|
|
|
|
| Finding | First checks |
|
|
|---|---|
|
|
| `reverse_shell` | Inspect parent process, fd table, remote peer, shell argv, and user. |
|
|
| `egress` | Confirm interpreter, destination IP, command line, and parent process. |
|
|
| `new_listener` | Identify owning process and whether the binary is package-owned. |
|
|
| `new_suid` | Check path, owner, package provenance, and whether it lives in a writable directory. |
|
|
| `persistence` | Diff the changed file and identify the modifying package or process if possible. |
|
|
| `fim_modified` | Verify package ownership, package checksum, and whether the change followed an upgrade. |
|
|
| `pkgdb_tamper` | Treat as high-confidence tampering until a legitimate package transaction explains it. |
|
|
| `pkg_signature_mismatch` | Treat as a potentially trojaned package-owned file. |
|
|
| `hidden_*` rootcheck alerts | Compare live tools, preserve evidence, and consider offline analysis. |
|
|
|
|
## Baseline Hygiene
|
|
|
|
Baselines are useful only if they represent a known-good state.
|
|
|
|
- Build baselines after installing Sentinel on a clean host.
|
|
- Rebuild listener/SUID baselines after intentionally adding long-running
|
|
services or privileged binaries.
|
|
- Use `fim-update` only after confirming changes are legitimate. Package hooks
|
|
already run it for normal package transactions.
|
|
- Keep a copy of important anchors off-box when possible.
|
|
|
|
## Suggested Hardening
|
|
|
|
- Keep the dashboard bound to Tailscale or loopback, not a public interface.
|
|
- Set `web_token` explicitly for fully read-only service operation.
|
|
- Enable at least one push backend.
|
|
- Run `watchdog` from another machine.
|
|
- Consider `chattr +i` for Sentinel binaries, systemd units, config, baselines,
|
|
and the pacman hook after setup.
|
|
- Keep the package cache populated if using signed-package verification.
|
|
- Do not disable package signature verification.
|
|
|
|
## Evidence Export
|
|
|
|
Until the incident command set exists, an evidence bundle is simply:
|
|
|
|
```bash
|
|
tar -C /var/log -czf /tmp/enodia-sentinel-evidence.tgz enodia-sentinel
|
|
```
|
|
|
|
That bundle contains events, snapshots, baselines, heartbeat state, package DB
|
|
anchor, and dashboard-readable JSON.
|