enodia-sentinal/docs/OPERATIONS.md

# Enodia Sentinel Operations Guide

This guide turns the current toolset into repeatable operator workflows. It is
deliberately practical: what to run, what to expect, and what to check next.

## First Install Checklist

1. Install and enable the service.

   ```bash
   sudo make install
   sudo make enable
   ```

2. Build the listener and SUID baselines.

   ```bash
   sudo enodia-sentinel baseline
   ```

3. Build the FIM baseline.

   ```bash
   sudo enodia-sentinel fim-baseline
   ```

4. Run a one-shot check.

   ```bash
   sudo enodia-sentinel check
   sudo enodia-sentinel rootcheck
   sudo enodia-sentinel pkgdb-check
   ```

5. Start the dashboard if desired.

   ```bash
   sudo systemctl enable --now enodia-sentinel-web
   ```

   Open the `https://...` URL from the service log. The default certificate is
   self-signed, so add a browser exception or configure `web_tls_cert` and
   `web_tls_key` with your own local/private CA certificate.

6. Configure one off-box notification or watchdog path. A local-only alerting
   path is not enough if the whole host goes silent.

## Routine Health Checks

Run these after upgrades, config edits, or suspicious behavior:

```bash
enodia-sentinel check
enodia-sentinel fim-check
enodia-sentinel pkgdb-check
enodia-sentinel pkgdb-verify --sample 100
enodia-sentinel rootcheck
enodia-sentinel posture check
enodia-sentinel triage
```

Expected healthy output:

- `check`: no alerts, or only expected tuned findings.
- `fim-check`: no changes against baseline after legitimate updates have been
  acknowledged.
- `pkgdb-check`: package DB consistent with anchor.
- `pkgdb-verify`: sampled files match signed cache packages.
- `rootcheck`: no hidden processes, processes hidden from `ps`, hidden
  modules/ports/raw/special-protocol sockets, sniffers, known rootkit modules,
  raw ICMP/SCTP-style channels, or unexplained kernel/module taint.
- `posture check`: no SSH/sudo/PATH/permission/signature hygiene findings, or
  only ones you have consciously accepted (e.g. password auth on a host that
  needs it).

The HTTPS dashboard shows the same posture findings under the Posture tab and
summarizes FIM/package anchors plus heartbeat freshness under the Integrity tab
for quick remote review.

For SSH/tmux operators, `enodia-sentinel tui` provides a terminal dashboard over
the same local read-only management state: status, recent alerts, incidents,
incident timelines, posture findings, integrity/watchdog state, event-rule
metadata, event tail, and dry-run response-plan previews. Use `:` for command
mode and `Tab` to complete TUI commands.

On desktop hosts the optional tray applet (`enodia-sentinel-tray`, `[tray]`
extra) wraps the same workflow: it drives the same `systemctl` units operators
use directly, so daemon start/stop/restart prompts for polkit authorization as
usual. The applet is purely a convenience frontend — it is not required for the
daemon to run and adds nothing to the core agent.

## Alert Workflow

When Sentinel fires:

1. Open the newest JSON and text snapshots in `/var/log/enodia-sentinel`.
2. Identify the signature, `sid`, affected PIDs, executable paths, remote peers,
   and parent process.
3. Run `enodia-sentinel triage` to separate known benign listener noise from
   findings that need review.
4. Build a read-only response plan from the grouped incident:

   ```bash
   enodia-sentinel incident list
   enodia-sentinel respond plan <incident-id>
   ```

   Review the plan before acting; Sentinel does not execute containment
   commands. CLI-generated plans are saved under
   `/var/log/enodia-sentinel/response-plans/` and indexed in
   `/var/log/enodia-sentinel/response-audit.log` for later review.
5. If the alert involves a package-owned binary, review the package recovery
   actions in the response plan and run the restore sequence only after the
   owning package and package source are trusted:

   ```bash
   pacman -Qo /path/to/file
   enodia-sentinel pkgdb-verify --sample 200
   pacman -S --overwrite '*' <package>
   enodia-sentinel fim-check --packages
   enodia-sentinel fim-update
   ```

6. If the alert involves hiding or tampering, run:

   ```bash
   enodia-sentinel rootcheck
   enodia-sentinel pkgdb-check
   ```

7. Preserve evidence before changing state:

   ```bash
   cp -a /var/log/enodia-sentinel /tmp/enodia-sentinel-evidence
   ```

8. Contain manually for now, following the matching playbook in
   [RUNBOOKS.md](RUNBOOKS.md). The response plan gives reviewed commands, but
   applying them remains an explicit operator action.

## Common Findings

| Finding | First checks |
|---|---|
| `reverse_shell` | Inspect parent process, fd table, remote peer, shell argv, and user. |
| `egress` | Confirm interpreter, destination IP, command line, and parent process. |
| `new_listener` | Identify owning process and whether the binary is package-owned. |
| `new_suid` | Check path, owner, package provenance, and whether it lives in a writable directory. |
| `persistence` | Diff the changed file and identify the modifying package or process if possible. |
| `fim_modified` | Verify package ownership, package checksum, and whether the change followed an upgrade. |
| `pkgdb_tamper` | Treat as high-confidence tampering until a legitimate package transaction explains it. |
| `pkg_signature_mismatch` | Treat as a potentially trojaned package-owned file. |
| `rootkit_*` / `kernel_tainted` alerts | Compare live tools, review module provenance, preserve evidence, and consider offline analysis. |

## Baseline Hygiene

Baselines are useful only if they represent a known-good state.

- Build baselines after installing Sentinel on a clean host.
- Rebuild listener/SUID baselines after intentionally adding long-running
  services or privileged binaries.
- Use `fim-update` only after confirming changes are legitimate. Package hooks
  already run it for normal package transactions.
- Keep a copy of important anchors off-box when possible.
- Prefer `baseline accept <kind> <target> --reason "..."` over rebuilding a whole
  baseline when only one item changed legitimately (an edited config, a new
  service port, an installed SUID helper). Unlike a rebuild, the acceptance is
  per-item, carries a recorded reason, is fingerprint-bound (FIM/package items
  re-alert if the file changes *again*), and can be time-boxed with `--expires`.
  Review outstanding acceptances with `baseline list`; it exits non-zero and
  marks rows `stale` when an accepted item has drifted again or its TTL lapsed.

## Suggested Hardening

- Keep the dashboard bound to Tailscale or loopback, not a public interface.
- Set `web_token` explicitly for fully read-only service operation.
- Keep HTTPS enabled. The built-in self-signed certificate is acceptable for a
  single host after you add an exception; use a private/local CA for cleaner
  browser trust.
- Enable at least one push backend.
- Run `watchdog` from another machine.
- Consider `chattr +i` for Sentinel binaries, systemd units, config, baselines,
  and the pacman hook after setup.
- Keep the package cache populated if using signed-package verification.
- Do not disable package signature verification.

## Evidence Export

Until the incident command set exists, an evidence bundle is simply:

```bash
tar -C /var/log -czf /tmp/enodia-sentinel-evidence.tgz enodia-sentinel
```

That bundle contains events, snapshots, baselines, heartbeat state, package DB
anchor, and dashboard-readable JSON.