enodia-sentinal/README.md

# Enodia Sentinel

A Linux host security platform: intrusion detection, file/package integrity,
anti-rootkit cross-checks, tamper-evidence, forensic snapshots, push alerts, and
a read-only operator console.

The current daemon continuously runs detectors over live system state —
processes, sockets, file descriptors, the SUID inventory, sensitive files, the
package database, and rootkit-hiding artifacts — then writes a detailed forensic
snapshot (text **and** JSON) with incident-response guidance the moment a known
attack signature appears.

Think of it as the security counterpart to a performance watchdog: instead of
"I/O pressure spiked, here's the kernel state," it's *"a shell just wired itself
to a socket — here's the process tree, the peer, what changed on disk, whether
the package database was tampered with, and what to do next."*

## Product direction

Enodia is becoming more than an IDS. The local detector is the first layer of a
broader host security platform:

| Function | Current capability | Direction |
|---|---|---|
| Detect | Poll detectors + eBPF exec rules | More event sources and correlation |
| Verify | FIM, package DB anchor, signed-package checks | External anchors and signed evidence |
| Investigate | Text/JSON snapshots, dashboard, triage | Incident timelines and evidence export |
| Respond | Human guidance today | Dry-run response plans and audited containment |
| Assure | Heartbeat, watchdog, self-integrity, rootcheck | Fleet health and attestation-ready anchors |

Project docs:

- [Specification](docs/SPECIFICATION.md) — product model, current scope, target
  platform shape, data model, and acceptance criteria.
- [Roadmap](docs/ROADMAP.md) — phased work from local sensor to incident,
  response, fleet, and assurance layers.
- [Operations guide](docs/OPERATIONS.md) — install checks, health checks, alert
  workflow, baseline hygiene, and evidence export.

> **Two implementations, on purpose.** The project began as a bash prototype
> (`src/sentinel.sh`, kept as the regression **oracle**) and was re-architected
> into a zero-dependency Python package with a unit-test suite, structured
> detectors, and JSON output. The bash version and the Python version share one
> red-team harness, so every signature is exercised against both.

## Why these detectors

Every detector keys on a behavior that is **cheap to observe** and **expensive
for an attacker to avoid** — the high-signal, low-false-positive heuristics real
EDRs are built on:

| Signature | What it catches | Why it's hard to evade |
|---|---|---|
| `reverse_shell` | An interpreter with a **network socket on fd 0/1/2** | Interactive shells get a pty and daemons get unix sockets — a *network* socket on stdio is `nc -e` / `bash -i >& /dev/tcp/...` |
| `ld_preload` | Non-empty `/etc/ld.so.preload`, or `LD_PRELOAD` into a writable dir | Injecting into processes needs the library to exist somewhere |
| `deleted_exe` | A process running from a **deleted / `memfd:`** binary | Fileless malware deletes its dropper; the kernel still names the inode `(deleted)` |
| `new_listener` | A listening port absent from the startup baseline | Bind shells/backdoors have to listen somewhere |
| `new_suid` | A new SUID/SGID binary (critical in a writable dir) | A SUID `/tmp` binary is a textbook privesc trick |
| `persistence` | Changes to cron, systemd units, `authorized_keys`, rc files | Persistence has to write somewhere that survives reboot |
| `egress` | An interpreter with an established connection to a public IP | C2 beacons and exfil have to phone home |

Every detection carries a stable **`sid`** and a **`classtype`** (à la
Snort/Suricata), so it can be referenced, tuned, and tracked across revisions.

## Event-driven detection (eBPF + a Snort-style rule engine)

Polling has a blind spot: a process that runs and exits between two sweeps is
invisible to it. The event layer closes that gap. An eBPF probe (loaded with
`bcc`) fires on every `execve` and hands each event to a **declarative rule
engine** — the host-event analogue of Snort matching packets:

```toml
# a rule is data, not code — sid, msg, classtype, severity + conditions
sid        = 100002
msg        = "Reverse-shell command pattern in execve arguments"
severity   = "CRITICAL"
classtype  = "c2-reverse-shell"
argv_regex = "/dev/(tcp|udp)/| -i\\b| -e\\b| pty\\.spawn"
```

Shipped rules cover fileless execution from world-writable dirs (`sid 100001`),
reverse-shell argv patterns (`100002`), web/DB services spawning a shell —
webshell/RCE (`100003`), and `curl|sh`-style ingress tool transfer (`100004`).
Operators add their own via `exec_rules_file` without touching code.

The layer is **fail-safe**: if `bcc`/root/BTF aren't available it logs the
reason and the daemon runs poll-only — a broken probe can never take detection
down. Lineage: the rule-driven engine + SIDs come from **Snort**; the host-IDS
framing (and the queued FIM / hidden-process checks) from **OSSEC**.

## Architecture

```
enodia_sentinel/
├── cli.py            run / check / baseline / list-detectors
├── daemon.py         sweep loop · cooldown dedup · backgrounded SUID scan
├── system.py         SystemState — one cached snapshot of /proc + ss per sweep
├── snapshot.py       forensic text+JSON capture · response guidance · retention
├── config.py         dataclass config (TOML + env overrides)
├── netutil.py        public-IP / CIDR logic (stdlib ipaddress)
├── alert.py          Alert / Severity (with Snort-style sid + classtype)
├── web.py            read-only dashboard: stdlib http server + JSON API + auth
├── static/           the self-contained dashboard SPA
├── fim.py            file integrity monitoring (hash baseline + pacman verify)
├── pkgdb.py          package-DB integrity + signed-package verification
├── rootcheck.py      anti-rootkit cross-view (hidden procs/modules/ports)
├── selfprotect.py    self-integrity footprint + dead-man's-switch heartbeat
├── triage.py         false-positive classification
├── provenance.py     package-ownership lookups (pacman/dpkg/rpm)
├── detectors/        poll detectors — one module per signature, each a pure
│                       function: detect(state, cfg) -> Iterable[Alert]
├── notify/           outbound push — ntfy / Pushover / webhook backends
└── events/           event-driven layer (eBPF)
    ├── bcc_source.py   real eBPF execve probe loaded via bcc
    ├── exec_event.py   the ExecEvent type
    ├── rules.py        Snort-style ExecRule engine + default rules
    └── monitor.py      runs the probe on a thread, routes events → rules
```

Two complementary detection paths feed one Alert → snapshot pipeline:

- **Poll** — every few seconds, sweep `/proc`/`ss` (catches anything lingering).
- **Event** — eBPF fires on every `execve`, matched against the rule engine
  (catches processes that exit *between* sweeps).

The loop is deliberately the same control flow as the bash prototype, but the
state lives in real objects:

```
every sample_interval seconds:
    state  = SystemState()                 # /proc + ss gathered once, cached
    alerts = run_all(detectors, state, cfg)
    fresh  = drop alerts still within cooldown
    if fresh:
        snapshot.capture(fresh)            # on a worker thread
```

Two design choices keep it fast and unobtrusive:

- **One `SystemState` per sweep.** Detectors read shared, cached `/proc`/`ss`
  data instead of each shelling out — a sweep costs ~200 ms regardless of how
  many detectors run.
- **The filesystem-wide SUID scan runs off the loop thread** on a slow cadence,
  so the multi-second walk never stalls live detection.

Everything in `SystemState` is **injectable**, which is what makes the detectors
unit-testable without root or a live system (see `tests/`).

## Quick start

```bash
sudo make install
sudo make enable          # start + enable the systemd service

# prove it works — in one terminal:
sudo tail -f /var/log/enodia-sentinel/events.log
# in another:
sentinel-redteam          # safe, self-cleaning attack simulations
```

You'll watch the drills trip the core poll signatures in real time, each
producing a `.log` + `.json` snapshot with response guidance.

Output lives in `/var/log/enodia-sentinel/`:
- `events.log` — one line per alert
- `alert-YYYYMMDD-HHMMSS.log` — human-readable forensic snapshot
- `alert-YYYYMMDD-HHMMSS.json` — same data, structured (SIEM-ready)

### Without installing

```bash
make test                                   # run the unit suite
python3 -m enodia_sentinel.cli baseline     # establish baselines
python3 -m enodia_sentinel.cli check        # run every detector once, print findings
```

No pip, no virtualenv, no dependencies — it's stdlib-only and installs as a
plain package directory plus a launcher wrapper.

## The red-team harness

`sentinel-redteam` is the demo and the integration test in one. It simulates
each threat with **safe, clearly-labeled stand-ins** (everything tagged
`enodia-drill`, auto-cleaned on exit), using a local Python TCP listener so no
traffic ever leaves the host:

```bash
sentinel-redteam --list                  # list drills
sentinel-redteam reverse_shell new_suid  # run specific ones
HOLD=30 sentinel-redteam                  # keep artifacts alive 30s
```

It never touches your real dotfiles or `/etc/ld.so.preload`; the LD_PRELOAD
drill only sets the env var on a throwaway process.

## Testing

```bash
make test          # stdlib unittest suite, no runtime deps
```

Detectors are pure functions over an injectable `SystemState`, so tests build
fake processes/sockets and assert on the alerts — no root, no `/proc`, no `ss`:

```python
proc  = FakeProc(pid=100, comm="bash", _stdio_inode=999)
sock  = Socket("ESTAB", "127.0.0.1:55", "9.9.9.9:443", 999, "bash", 100)
state = SystemState(processes=[proc], sockets=[sock])
assert list(reverse_shell.detect(state, Config()))[0].signature == "reverse_shell"
```

## Configuration

Edit `/etc/enodia-sentinel.toml`, then `sudo systemctl restart
enodia-sentinel.service`. Every key is optional. Highlights:

| Key | Default | Purpose |
|---|---|---|
| `sample_interval` | 4 | seconds between sweeps |
| `cooldown` | 60 | min seconds before re-alerting a signature |
| `detectors` | all 7 | the enabled detector list |
| `interpreters` | bash sh … | process names treated as shells |
| `egress_allow_cidrs` | [] | trusted public ranges (won't trip egress) |
| `suid_hot_dirs` | /tmp … | dirs where a SUID binary is CRITICAL |
| `suid_scan_extra_dirs` | /tmp … | writable mounts always scanned (tmpfs-safe) |
| `capture_execve_bpftrace` | false | add a bpftrace execve trace to snapshots |
| `notify_users` | [] | desktop notify-send targets |
| `pkgdb_pkgverify` | false | verify on-disk files against signed cache packages |
| `pkgdb_pkgverify_sample` | 40 | packages verified per pass (rotates over time) |
| `rootcheck_enabled` | true | run the anti-rootkit cross-view sweep |
| `rootcheck_interval` | 300 | seconds between cross-view sweeps |

## Web dashboard

A read-only console, served by the stdlib `http.server` (no Flask, no JS
framework, no CDN — one self-contained page):

```bash
enodia-sentinel web                      # serves on the Tailscale IP by default
# or as a service:
sudo systemctl enable --now enodia-sentinel-web
```

- **Bound to your Tailscale interface** by default (auto-detected), so it's
  reachable from your phone/laptop on the tailnet but not the LAN or internet.
- **Bearer-token auth** (constant-time check); the token is auto-generated and
  saved on first run and printed in the startup line. Open
  `http://<tailscale-ip>:8787/?token=…`.
- **Read-only**: severity cards, a live alert list, and the full forensic
  snapshot per alert. No actions, no writes — minimal attack surface for
  sensitive data. JSON API at `/api/status`, `/api/alerts`, `/api/alerts/<id>`,
  `/api/events`.

## Phone push notifications

When an alert at/above `notify_min_severity` fires, Sentinel pushes to whichever
backends you've configured (all via stdlib `urllib`, no SDKs):

| Backend | Enable by setting | Notes |
|---|---|---|
| **ntfy** | `notify_ntfy_url` + `notify_ntfy_topic` | open-source, self-hostable, free apps |
| **Pushover** | `notify_pushover_token` + `_user` | polished, reliable |
| **Webhook** | `notify_webhook_url` | generic JSON POST (Discord/Slack/your own) |

Severity maps to each service's priority (a CRITICAL is an urgent ntfy push / a
high-priority Pushover). Sends happen on worker threads and swallow their own
errors — a flaky notifier never stalls detection.

```toml
notify_min_severity = "HIGH"
notify_ntfy_url   = "https://ntfy.sh"
notify_ntfy_topic = "enodia-7Hq2x"      # keep this secret — it's the access control
```

## Tamper-evidence & self-protection

An attacker's first move against a sensor is to disable or blind it — and on a
box where they have root, *any* purely-local defense is ultimately defeatable
(they share your privileges). Sentinel doesn't pretend otherwise. The goal is to
make tampering **evident** by anchoring trust where the attacker has less
control, and to make going silent *loud*.

**Package-DB integrity.** `pacman -Qkk` trusts the local package database — so a
root attacker can modify `/usr/bin/ssh` *and* rewrite its stored checksum, and
verification passes. Sentinel guards the DB itself: a legitimate change only
happens during a logged transaction, so it anchors a fingerprint of
`/var/lib/pacman/local` (refreshed *only* by the pacman hook) and cross-checks
`pacman.log`. A DB change with **no corresponding transaction** is flagged
`pkgdb_tamper` (CRITICAL, `sid 100021`) — directly catching the "overwrite the
hashes" attack.

**Self-integrity.** Sentinel's own binaries, config, systemd units, and pacman
hook are always in the FIM watch set, so tampering with the watchdog trips the
watchdog.

**Dead-man's switch.** The daemon writes a heartbeat every loop; the dashboard
exposes its age. Run an external watcher on another host (over Tailscale) and
*silence becomes the alarm*:

```bash
# on a SEPARATE machine — alerts you if the sensor or the whole box goes dark
enodia-sentinel watchdog --url http://100.x.x.x:8787 --token <T> --max-age 120
```

**Hardening (the layers beyond on-box detection):**

1. **Immutability** — `chattr +i` Sentinel's binaries, config, baselines, and
   the pacman hook so even root must visibly clear the flag first.
2. ✅ **Cryptographic anchor (done)** — verify on-disk files against the *signed*
   package in the cache rather than the mutable DB; a maintainer's PGP signature
   is a root of trust the attacker doesn't hold. See *Signed-package verification*
   below.
3. **External anchor (next)** — mirror the DB/FIM fingerprints off-box so a
   local attacker can't refresh the anchor to cover their tracks.

> On "anti-rootkit": the robust version isn't stealth (fragile, and a genuine
> dual-use rootkit technique) — it's tamper-*evidence*. Don't hide; be
> un-hideable. The durable defenses move the trust anchor below or outside the
> attacker: signatures, external monitors, and ultimately the kernel (the eBPF
> roadmap) rather than userland the attacker can rewrite.

### Signed-package verification (the independent anchor)

Layer 1 (above) catches an *out-of-band* DB edit. But a root attacker can do the
full job: modify `/usr/bin/sshd`, rewrite its hash in the local DB, **and** run
`fim-update` to re-anchor — defeating both `pacman -Qkk` and the DB-fingerprint
check, because every reference they're checked against is one they can rewrite.

The one reference they *can't* forge is the distro's signing key. Packages in
the cache are signed, and each carries a `.MTREE` manifest of per-file SHA-256
hashes. Layer 2 extracts that manifest from the **cached package** and compares
the on-disk files to it — a reference independent of the local DB:

```bash
enodia-sentinel pkgdb-verify              # verify a rotating sample of packages
enodia-sentinel pkgdb-verify --sample 200 # verify more per run
```

A divergence is `pkg_signature_mismatch` (CRITICAL, `sid 100027`) — a trojaned
binary that a rewritten checksum DB would have hidden. It also flags a
`SigLevel` downgrade in `pacman.conf` (`pacman_siglevel_disabled`, CRITICAL,
`sid 100026`), since disabling signature checking is how an attacker would slip
an unsigned package past the anchor in the first place.

Verifying every package each pass is expensive (untar + hash every file), so it
runs on its own slow cadence (`pkgdb_pkgverify_interval`) and checks a **rotating
sample** (`pkgdb_pkgverify_sample`) per pass — over enough passes the whole
installed set is covered, and any single mismatch fires immediately. It's
off by default (needs the package cache populated); enable with `pkgdb_pkgverify
= true`.

## Anti-rootkit (cross-view detection)

A rootkit hides by lying to one view of the system — but the technique that
hides it is also how you catch it: **ask the same question two different ways and
compare the answers.** A discrepancy is the hiding artifact.

| Cross-check | Hidden thing it surfaces | `sid` |
|---|---|---|
| `kill(pid, 0)` for every PID vs the `/proc` listing | a process the kernel schedules but `/proc` omits | 100022 |
| `/sys/module` (initstate=live) vs `/proc/modules` | a loaded LKM hidden from the module list | 100023 |
| `/proc/net/tcp` vs `ss` | a listening port a hooked `ss` won't report | 100024 |
| `/sys/class/net/*/flags` | an interface in promiscuous mode (a sniffer) | 100025 |

```bash
enodia-sentinel rootcheck     # one-shot cross-view scan
```

The daemon runs it on a slow background cadence (`rootcheck_interval`) and routes
any finding into the normal alert/snapshot/push pipeline. **Honest limits:** this
runs in user space, so it reliably catches userland (`LD_PRELOAD`) rootkits and
the common `/proc`-hiding LKMs, but a kernel rootkit that hooks *every* path
consistently can still evade it. It raises the bar and catches the common cases;
it is not a guarantee against a bespoke ring-0 implant — pair it with the off-box
dead-man's switch. (Lineage: OSSEC's rootcheck / `chkrootkit`'s cross-view idea.)

## File integrity monitoring (Tripwire-style)

Detects tampering with binaries and critical configs by **content hash**, so it
catches a malicious swap even when the timestamp is preserved (the weakness of
mtime-based checks). Two engines, split by who owns the file:

- **Package verification** — `pacman -Qkk` checks package-owned binaries against
  the distro's *own signed checksums*. No baseline to maintain, and it's
  implicitly current because the package DB updates on every `pacman -Syu`. A
  trojaned `/usr/bin/ssh` surfaces as a checksum mismatch (CRITICAL, `sid 100020`).
- **Hash baseline** — a SHA-256 baseline of the files pacman *doesn't* track
  (`/usr/local`, `/etc` configs, systemd units, SSH keys). A pacman
  `PostTransaction` **hook** runs `fim-update` after every upgrade, so legitimate
  package changes never alert — no manual `tripwire --update` ritual.

```bash
enodia-sentinel fim-baseline          # establish the baseline
enodia-sentinel fim-check             # report changes vs baseline
enodia-sentinel fim-check --packages  # also verify package-owned files
```

The baseline is **only** refreshed by `fim-update` / the pacman hook — a flagged
change stays flagged until you acknowledge it, exactly like Tripwire. The daemon
runs the scan on a slow background cadence (`fim_scan_interval`) and routes any
change into the normal alert/snapshot/push pipeline (`fim_modified` `sid 100017`,
`fim_added` `100018`, `fim_removed` `100019`).

## False positives & triage

A host IDS that cries wolf gets ignored, so Sentinel ships explicit tooling to
separate benign noise from real findings — built on **provenance**: a binary
shipped by your package manager (`pacman`/`dpkg`/`rpm`) is overwhelmingly likely
to be legitimate (the same idea behind OSSEC's rootcheck and AIDE).

```bash
enodia-sentinel triage     # classify captured alerts, suggest allowlist entries
```

```
12 distinct detections — 11 likely false-positive, 1 to review.

[FP  ] new_listener   x99  listener binary is package-owned (qbittorrent)
[FP  ] new_listener   x1   loopback-only listener (not externally reachable)
[REVIEW] new_listener x1   unrecognized listener *:1740 (?)
...
To suppress the false positives, add to your config:
  # listener_allow_comms += "qbittorrent"
```

Triage is deliberately conservative: `reverse_shell`, `egress`, and the eBPF
exec rules are **always** flagged review (provenance can't clear a network
shell), and any listener it *can't* attribute to a process is reviewed rather
than cleared. Suppression is never automatic — you choose what to allowlist.

Knobs to quiet known-good activity:

| Config | Effect |
|---|---|
| `listener_allow_comms` | never alert on listeners owned by these apps |
| `listener_allow_ports` | never alert on these ports |
| `suppress_package_owned_listeners` | drop `new_listener` when the binary is package-owned (best single knob for a desktop/seedbox) |
| `egress_allow_cidrs` | trusted public ranges for the egress detector |
| `suid_hot_dirs` / `exec_rules_file` | tune SUID criticality / add custom exec rules |

## Security model

Sentinel runs as root because it must read every process's `/proc`, the full
socket table, and root-owned files like `authorized_keys`. The systemd unit
constrains that power: `ProtectSystem=strict` with the log dir as the only
writable path, `ProtectHome=read-only`, `NoNewPrivileges`,
`MemoryDenyWriteExecute`, `RestrictNamespaces`, and a minimal capability set
(`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`). It only ever **reads** the system and
**writes** to its own log directory.

## Enabling the eBPF monitor

The event layer needs `python-bpfcc` and privileges the hardened unit
deliberately withholds (bcc JIT-compiles its programs, so it needs write+exec
memory and `CAP_BPF`/`CAP_PERFMON`/`CAP_SYS_ADMIN`). Under the default unit the
monitor simply fails closed and the daemon runs poll-only. To turn it on:

```bash
sudo pacman -S python-bpfcc
sudo install -Dm644 systemd/enodia-sentinel-ebpf.conf \
    /etc/systemd/system/enodia-sentinel.service.d/ebpf.conf
sudo systemctl daemon-reload && sudo systemctl restart enodia-sentinel
# confirm:  grep 'eBPF exec monitor' /var/log/enodia-sentinel/events.log
```

The drop-in relaxes `MemoryDenyWriteExecute` and widens the capability set — a
conscious tradeoff documented in the file itself.

## Roadmap

The short version: Enodia grows from local HIDS into a host security platform:
incident grouping, posture checks, response planning, richer eBPF telemetry,
fleet health, external anchors, and eventually attestation-ready assurance.

See [docs/ROADMAP.md](docs/ROADMAP.md) for the release tracks and phased plan.
The polling daemon isn't throwaway — it's the **oracle**: every signature is a
test case the event layer must reproduce, and `sentinel-redteam` is the shared
regression suite for both.

## Project status

v0.7 — closes the tamper-evidence loop with the **independent anchor**:
signed-package verification (compares on-disk files to the `.MTREE` in the signed
cache package, surviving a rewritten checksum DB) plus a `SigLevel`-downgrade
check, and an **anti-rootkit cross-view** layer (hidden processes, modules,
ports, and promiscuous interfaces — each caught by asking the same question two
ways and diffing the answers).

v0.6 — adds **tamper-evidence**: out-of-band package-DB integrity
(catches rewritten checksums), self-integrity of Sentinel's own footprint, and a
dead-man's-switch heartbeat with an external watchdog.

v0.5 — adds **file integrity monitoring** (SHA-256 baseline + `pacman -Qkk`
verification, auto-refreshed by a pacman hook) and **false-positive triage**
via package-ownership provenance.

v0.4 — adds a read-only **web dashboard** (stdlib server, Tailscale-bound,
token-auth) and **phone push** (ntfy / Pushover / webhook), both zero-dependency.

v0.3 — adds the event-driven **eBPF layer**: a real `bcc` execve probe feeding a
Snort-style declarative rule engine (4 default rules), stable signature IDs +
classtypes on every detection, fail-safe degradation to poll-only, and an
opt-in hardening drop-in. Inspired by Snort (rule engine, SIDs) and OSSEC (HIDS
framing; FIM + hidden-process checks are next).

v0.2 — Python re-architecture of the bash prototype: 7 detectors, text+JSON
forensic snapshots, backgrounded SUID scanning, 25-test unit suite, red-team
harness, hardened systemd unit, Arch packaging. Zero runtime dependencies.
Built and tested on Arch Linux.

## License

GPL-3.0-or-later — see [LICENSE](LICENSE).