enodia-sentinal/README.md

# Enodia Sentinel

A host intrusion-detection daemon for Linux. It continuously runs a set of
detectors over live system state — processes, sockets, file descriptors, the
SUID inventory, and sensitive files — and writes a detailed forensic snapshot
(text **and** JSON) with incident-response guidance the moment a known attack
signature appears.

Think of it as the security counterpart to a performance watchdog: instead of
"I/O pressure spiked, here's the kernel state," it's *"a shell just wired itself
to a socket — here's the process tree, the peer, and what to do about it."*

> **Two implementations, on purpose.** The project began as a bash prototype
> (`src/sentinel.sh`, kept as the regression **oracle**) and was re-architected
> into a zero-dependency Python package with a unit-test suite, structured
> detectors, and JSON output. The bash version and the Python version share one
> red-team harness, so every signature is exercised against both.

## Why these detectors

Every detector keys on a behavior that is **cheap to observe** and **expensive
for an attacker to avoid** — the high-signal, low-false-positive heuristics real
EDRs are built on:

| Signature | What it catches | Why it's hard to evade |
|---|---|---|
| `reverse_shell` | An interpreter with a **network socket on fd 0/1/2** | Interactive shells get a pty and daemons get unix sockets — a *network* socket on stdio is `nc -e` / `bash -i >& /dev/tcp/...` |
| `ld_preload` | Non-empty `/etc/ld.so.preload`, or `LD_PRELOAD` into a writable dir | Injecting into processes needs the library to exist somewhere |
| `deleted_exe` | A process running from a **deleted / `memfd:`** binary | Fileless malware deletes its dropper; the kernel still names the inode `(deleted)` |
| `new_listener` | A listening port absent from the startup baseline | Bind shells/backdoors have to listen somewhere |
| `new_suid` | A new SUID/SGID binary (critical in a writable dir) | A SUID `/tmp` binary is a textbook privesc trick |
| `persistence` | Changes to cron, systemd units, `authorized_keys`, rc files | Persistence has to write somewhere that survives reboot |
| `egress` | An interpreter with an established connection to a public IP | C2 beacons and exfil have to phone home |

Every detection carries a stable **`sid`** and a **`classtype`** (à la
Snort/Suricata), so it can be referenced, tuned, and tracked across revisions.

## Event-driven detection (eBPF + a Snort-style rule engine)

Polling has a blind spot: a process that runs and exits between two sweeps is
invisible to it. The event layer closes that gap. An eBPF probe (loaded with
`bcc`) fires on every `execve` and hands each event to a **declarative rule
engine** — the host-event analogue of Snort matching packets:

```toml
# a rule is data, not code — sid, msg, classtype, severity + conditions
sid        = 100002
msg        = "Reverse-shell command pattern in execve arguments"
severity   = "CRITICAL"
classtype  = "c2-reverse-shell"
argv_regex = "/dev/(tcp|udp)/| -i\\b| -e\\b| pty\\.spawn"
```

Shipped rules cover fileless execution from world-writable dirs (`sid 100001`),
reverse-shell argv patterns (`100002`), web/DB services spawning a shell —
webshell/RCE (`100003`), and `curl|sh`-style ingress tool transfer (`100004`).
Operators add their own via `exec_rules_file` without touching code.

The layer is **fail-safe**: if `bcc`/root/BTF aren't available it logs the
reason and the daemon runs poll-only — a broken probe can never take detection
down. Lineage: the rule-driven engine + SIDs come from **Snort**; the host-IDS
framing (and the queued FIM / hidden-process checks) from **OSSEC**.

## Architecture

```
enodia_sentinel/
├── cli.py            run / check / baseline / list-detectors
├── daemon.py         sweep loop · cooldown dedup · backgrounded SUID scan
├── system.py         SystemState — one cached snapshot of /proc + ss per sweep
├── snapshot.py       forensic text+JSON capture · response guidance · retention
├── config.py         dataclass config (TOML + env overrides)
├── netutil.py        public-IP / CIDR logic (stdlib ipaddress)
├── alert.py          Alert / Severity (with Snort-style sid + classtype)
├── web.py            read-only dashboard: stdlib http server + JSON API + auth
├── static/           the self-contained dashboard SPA
├── fim.py            file integrity monitoring (hash baseline + pacman verify)
├── triage.py         false-positive classification
├── provenance.py     package-ownership lookups (pacman/dpkg/rpm)
├── detectors/        poll detectors — one module per signature, each a pure
│                       function: detect(state, cfg) -> Iterable[Alert]
├── notify/           outbound push — ntfy / Pushover / webhook backends
└── events/           event-driven layer (eBPF)
    ├── bcc_source.py   real eBPF execve probe loaded via bcc
    ├── exec_event.py   the ExecEvent type
    ├── rules.py        Snort-style ExecRule engine + default rules
    └── monitor.py      runs the probe on a thread, routes events → rules
```

Two complementary detection paths feed one Alert → snapshot pipeline:

- **Poll** — every few seconds, sweep `/proc`/`ss` (catches anything lingering).
- **Event** — eBPF fires on every `execve`, matched against the rule engine
  (catches processes that exit *between* sweeps).

The loop is deliberately the same control flow as the bash prototype, but the
state lives in real objects:

```
every sample_interval seconds:
    state  = SystemState()                 # /proc + ss gathered once, cached
    alerts = run_all(detectors, state, cfg)
    fresh  = drop alerts still within cooldown
    if fresh:
        snapshot.capture(fresh)            # on a worker thread
```

Two design choices keep it fast and unobtrusive:

- **One `SystemState` per sweep.** Detectors read shared, cached `/proc`/`ss`
  data instead of each shelling out — a sweep costs ~200 ms regardless of how
  many detectors run.
- **The filesystem-wide SUID scan runs off the loop thread** on a slow cadence,
  so the multi-second walk never stalls live detection.

Everything in `SystemState` is **injectable**, which is what makes the detectors
unit-testable without root or a live system (see `tests/`).

## Quick start

```bash
sudo make install
sudo make enable          # start + enable the systemd service

# prove it works — in one terminal:
sudo tail -f /var/log/enodia-sentinel/events.log
# in another:
sentinel-redteam          # safe, self-cleaning attack simulations
```

You'll watch the drills trip `reverse_shell`, `ld_preload`, `deleted_exe`,
`new_listener`, and `new_suid` in real time, each producing a `.log` + `.json`
snapshot with response guidance.

Output lives in `/var/log/enodia-sentinel/`:
- `events.log` — one line per alert
- `alert-YYYYMMDD-HHMMSS.log` — human-readable forensic snapshot
- `alert-YYYYMMDD-HHMMSS.json` — same data, structured (SIEM-ready)

### Without installing

```bash
make test                                   # run the unit suite
python3 -m enodia_sentinel.cli baseline     # establish baselines
python3 -m enodia_sentinel.cli check        # run every detector once, print findings
```

No pip, no virtualenv, no dependencies — it's stdlib-only and installs as a
plain package directory plus a launcher wrapper.

## The red-team harness

`sentinel-redteam` is the demo and the integration test in one. It simulates
each threat with **safe, clearly-labeled stand-ins** (everything tagged
`enodia-drill`, auto-cleaned on exit), using a local Python TCP listener so no
traffic ever leaves the host:

```bash
sentinel-redteam --list                  # list drills
sentinel-redteam reverse_shell new_suid  # run specific ones
HOLD=30 sentinel-redteam                  # keep artifacts alive 30s
```

It never touches your real dotfiles or `/etc/ld.so.preload`; the LD_PRELOAD
drill only sets the env var on a throwaway process.

## Testing

```bash
make test          # 25 unit tests, stdlib unittest, no deps
```

Detectors are pure functions over an injectable `SystemState`, so tests build
fake processes/sockets and assert on the alerts — no root, no `/proc`, no `ss`:

```python
proc  = FakeProc(pid=100, comm="bash", _stdio_inode=999)
sock  = Socket("ESTAB", "127.0.0.1:55", "9.9.9.9:443", 999, "bash", 100)
state = SystemState(processes=[proc], sockets=[sock])
assert list(reverse_shell.detect(state, Config()))[0].signature == "reverse_shell"
```

## Configuration

Edit `/etc/enodia-sentinel.toml`, then `sudo systemctl restart
enodia-sentinel.service`. Every key is optional. Highlights:

| Key | Default | Purpose |
|---|---|---|
| `sample_interval` | 4 | seconds between sweeps |
| `cooldown` | 60 | min seconds before re-alerting a signature |
| `detectors` | all 7 | the enabled detector list |
| `interpreters` | bash sh … | process names treated as shells |
| `egress_allow_cidrs` | [] | trusted public ranges (won't trip egress) |
| `suid_hot_dirs` | /tmp … | dirs where a SUID binary is CRITICAL |
| `suid_scan_extra_dirs` | /tmp … | writable mounts always scanned (tmpfs-safe) |
| `capture_execve_bpftrace` | false | add a bpftrace execve trace to snapshots |
| `notify_users` | [] | desktop notify-send targets |

## Web dashboard

A read-only console, served by the stdlib `http.server` (no Flask, no JS
framework, no CDN — one self-contained page):

```bash
enodia-sentinel web                      # serves on the Tailscale IP by default
# or as a service:
sudo systemctl enable --now enodia-sentinel-web
```

- **Bound to your Tailscale interface** by default (auto-detected), so it's
  reachable from your phone/laptop on the tailnet but not the LAN or internet.
- **Bearer-token auth** (constant-time check); the token is auto-generated and
  saved on first run and printed in the startup line. Open
  `http://<tailscale-ip>:8787/?token=…`.
- **Read-only**: severity cards, a live alert list, and the full forensic
  snapshot per alert. No actions, no writes — minimal attack surface for
  sensitive data. JSON API at `/api/status`, `/api/alerts`, `/api/alerts/<id>`,
  `/api/events`.

## Phone push notifications

When an alert at/above `notify_min_severity` fires, Sentinel pushes to whichever
backends you've configured (all via stdlib `urllib`, no SDKs):

| Backend | Enable by setting | Notes |
|---|---|---|
| **ntfy** | `notify_ntfy_url` + `notify_ntfy_topic` | open-source, self-hostable, free apps |
| **Pushover** | `notify_pushover_token` + `_user` | polished, reliable |
| **Webhook** | `notify_webhook_url` | generic JSON POST (Discord/Slack/your own) |

Severity maps to each service's priority (a CRITICAL is an urgent ntfy push / a
high-priority Pushover). Sends happen on worker threads and swallow their own
errors — a flaky notifier never stalls detection.

```toml
notify_min_severity = "HIGH"
notify_ntfy_url   = "https://ntfy.sh"
notify_ntfy_topic = "enodia-7Hq2x"      # keep this secret — it's the access control
```

## File integrity monitoring (Tripwire-style)

Detects tampering with binaries and critical configs by **content hash**, so it
catches a malicious swap even when the timestamp is preserved (the weakness of
mtime-based checks). Two engines, split by who owns the file:

- **Package verification** — `pacman -Qkk` checks package-owned binaries against
  the distro's *own signed checksums*. No baseline to maintain, and it's
  implicitly current because the package DB updates on every `pacman -Syu`. A
  trojaned `/usr/bin/ssh` surfaces as a checksum mismatch (CRITICAL, `sid 100020`).
- **Hash baseline** — a SHA-256 baseline of the files pacman *doesn't* track
  (`/usr/local`, `/etc` configs, systemd units, SSH keys). A pacman
  `PostTransaction` **hook** runs `fim-update` after every upgrade, so legitimate
  package changes never alert — no manual `tripwire --update` ritual.

```bash
enodia-sentinel fim-baseline          # establish the baseline
enodia-sentinel fim-check             # report changes vs baseline
enodia-sentinel fim-check --packages  # also verify package-owned files
```

The baseline is **only** refreshed by `fim-update` / the pacman hook — a flagged
change stays flagged until you acknowledge it, exactly like Tripwire. The daemon
runs the scan on a slow background cadence (`fim_scan_interval`) and routes any
change into the normal alert/snapshot/push pipeline (`fim_modified` `sid 100017`,
`fim_added` `100018`, `fim_removed` `100019`).

## False positives & triage

A host IDS that cries wolf gets ignored, so Sentinel ships explicit tooling to
separate benign noise from real findings — built on **provenance**: a binary
shipped by your package manager (`pacman`/`dpkg`/`rpm`) is overwhelmingly likely
to be legitimate (the same idea behind OSSEC's rootcheck and AIDE).

```bash
enodia-sentinel triage     # classify captured alerts, suggest allowlist entries
```

```
12 distinct detections — 11 likely false-positive, 1 to review.

[FP  ] new_listener   x99  listener binary is package-owned (qbittorrent)
[FP  ] new_listener   x1   loopback-only listener (not externally reachable)
[REVIEW] new_listener x1   unrecognized listener *:1740 (?)
...
To suppress the false positives, add to your config:
  # listener_allow_comms += "qbittorrent"
```

Triage is deliberately conservative: `reverse_shell`, `egress`, and the eBPF
exec rules are **always** flagged review (provenance can't clear a network
shell), and any listener it *can't* attribute to a process is reviewed rather
than cleared. Suppression is never automatic — you choose what to allowlist.

Knobs to quiet known-good activity:

| Config | Effect |
|---|---|
| `listener_allow_comms` | never alert on listeners owned by these apps |
| `listener_allow_ports` | never alert on these ports |
| `suppress_package_owned_listeners` | drop `new_listener` when the binary is package-owned (best single knob for a desktop/seedbox) |
| `egress_allow_cidrs` | trusted public ranges for the egress detector |
| `suid_hot_dirs` / `exec_rules_file` | tune SUID criticality / add custom exec rules |

## Security model

Sentinel runs as root because it must read every process's `/proc`, the full
socket table, and root-owned files like `authorized_keys`. The systemd unit
constrains that power: `ProtectSystem=strict` with the log dir as the only
writable path, `ProtectHome=read-only`, `NoNewPrivileges`,
`MemoryDenyWriteExecute`, `RestrictNamespaces`, and a minimal capability set
(`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`). It only ever **reads** the system and
**writes** to its own log directory.

## Enabling the eBPF monitor

The event layer needs `python-bpfcc` and privileges the hardened unit
deliberately withholds (bcc JIT-compiles its programs, so it needs write+exec
memory and `CAP_BPF`/`CAP_PERFMON`/`CAP_SYS_ADMIN`). Under the default unit the
monitor simply fails closed and the daemon runs poll-only. To turn it on:

```bash
sudo pacman -S python-bpfcc
sudo install -Dm644 systemd/enodia-sentinel-ebpf.conf \
    /etc/systemd/system/enodia-sentinel.service.d/ebpf.conf
sudo systemctl daemon-reload && sudo systemctl restart enodia-sentinel
# confirm:  grep 'eBPF exec monitor' /var/log/enodia-sentinel/events.log
```

The drop-in relaxes `MemoryDenyWriteExecute` and widens the capability set — a
conscious tradeoff documented in the file itself.

## Roadmap

1. **bpftrace tracepoints (optional)** — `capture_execve_bpftrace = true` adds a
   live `execve` trace to each snapshot.
2. ✅ **Event-driven `execve` detection (done)** — a real eBPF probe via bcc,
   feeding a Snort-style rule engine, so a short-lived process can't slip
   between sweeps.
3. **More event sources** — `tcp_connect` (event-driven egress) and
   `security_bprm_check` (LSM) probes, plus per-process lineage tracking.
4. **A libbpf + CO-RE agent (Go or Rust)** — the production EDR core: ring-buffer
   event streaming, tamper resistance, no runtime compiler.

The polling daemon isn't throwaway — it's the **oracle**: every signature is a
test case the event layer must reproduce, and `sentinel-redteam` is the shared
regression suite for both.

## Project status

v0.5 — adds **file integrity monitoring** (SHA-256 baseline + `pacman -Qkk`
verification, auto-refreshed by a pacman hook) and **false-positive triage**
via package-ownership provenance.

v0.4 — adds a read-only **web dashboard** (stdlib server, Tailscale-bound,
token-auth) and **phone push** (ntfy / Pushover / webhook), both zero-dependency.

v0.3 — adds the event-driven **eBPF layer**: a real `bcc` execve probe feeding a
Snort-style declarative rule engine (4 default rules), stable signature IDs +
classtypes on every detection, fail-safe degradation to poll-only, and an
opt-in hardening drop-in. Inspired by Snort (rule engine, SIDs) and OSSEC (HIDS
framing; FIM + hidden-process checks are next).

v0.2 — Python re-architecture of the bash prototype: 7 detectors, text+JSON
forensic snapshots, backgrounded SUID scanning, 25-test unit suite, red-team
harness, hardened systemd unit, Arch packaging. Zero runtime dependencies.
Built and tested on Arch Linux.

## License

GPL-3.0-or-later — see [LICENSE](LICENSE).