diff --git a/docs/behavior/01-sweep-and-state.md b/docs/behavior/01-sweep-and-state.md new file mode 100644 index 0000000..da400d7 --- /dev/null +++ b/docs/behavior/01-sweep-and-state.md @@ -0,0 +1,355 @@ + +# 1. Sweep, System State, and Baseline Lifecycle + +Scope: how the daemon gathers one view of the host, runs poll detectors over +it, and manages the baselines that the baseline-diff detectors +(`first_seen`, `new_listener`, `new_suid`, `persistence`) depend on. This is +the substrate every other detection subsystem runs on top of. + +Source: `system.py`, `daemon.py`, `alert.py`, `detectors/__init__.py`, +`config.py`. + +--- + +## 1.1 The alert value + +`alert.py`. Every detection is an immutable `Alert`: + +| Field | Type | Meaning | +|-------|------|---------| +| `severity` | `Severity` enum | `MEDIUM=1`, `HIGH=2`, `CRITICAL=3`; `str()` yields the name | +| `signature` | str | human rule name, e.g. `new_listener` | +| `key` | str | **stable dedup identity**, e.g. `lis:31337/nc`; drives cooldown | +| `detail` | str | one-line human description | +| `pids` | tuple[int, …] | processes the snapshot should deep-dive (default empty) | +| `sid` | int | stable Snort-style signature id (default `0`) | +| `classtype` | str | category (default `uncategorized`) | + +`Alert.to_dict()` is the serialization contract — key order: +`sid, severity, signature, classtype, key, detail, pids` (with `severity` as +the string name and `pids` as a list). A port must preserve `key` values +exactly: cooldown dedup is keyed on them, so a differing key changes +re-fire behavior. + +--- + +## 1.2 System state capture + +`SystemState` (`system.py`) is **one sweep's cached view**, built once and +shared by all detectors. It is fully injectable: constructing it with +`processes=`/`sockets=` bypasses live collection (how tests and the parity +harness avoid `/proc` and `ss`). + +### 1.2.1 Processes + +`SystemState.processes` enumerates `/proc`, keeping entries whose name +`isdigit()`, each wrapped in a `Process(pid)`. Every `Process` field is a +`cached_property` read lazily from `/proc//…` and **fail-open** (any +`OSError` → empty value), because a process can exit mid-sweep: + +| Field | Source | Notes | +|-------|--------|-------| +| `comm` | `comm` | stripped | +| `cmdline` | `cmdline` | NUL→space, stripped | +| `exe` | readlink `exe` | `""` if unreadable | +| `cwd` | readlink `cwd` | | +| `environ` | `environ` | NUL-split `k=v` → dict | +| `fd_targets` | `fd/` | dict `fd(str) → link target`, fds sorted numerically | +| `memory_maps` | `maps` | see parser below | +| `status` | `status` | `key → value` (value stripped) | +| `ppid` | `status[PPid]` | `0` on parse failure | +| `uid` | `status[Uid]` field 0 | `0` if absent | + +`stdio_socket_inode()` returns the socket inode if fd 0, 1, or 2 links to +`socket:[…]` — the reverse-shell tell — else `None`. + +`parse_memory_maps(text)` splits each line `maxsplit=5`; needs ≥5 fields; +address `start-end` parsed as hex (skip line if no `-` or non-hex); `path` is +field 5 when present else `""`. Emits `MemoryMap(start, end, perms, path)`. +**The port must preserve `maxsplit=5` semantics** so pathnames containing +spaces stay intact. + +### 1.2.2 Sockets + +`SystemState.sockets` runs `ss -H` **once per protocol family**, in this order, +tagging each result with `kind`: + +``` +-tanep tcp -uanep udp -wanep raw -Sanep sctp -danep dccp +-0anep packet -Manep mptcp --tipc -anep tipc --xdp -anep xdp +--vsock -anep vsock +``` + +Each `ss` call is fail-open (`OSError`/`SubprocessError`/10s timeout → `""`), +so an unsupported family drops only its own rows. `_parse_ss` per line: + +- Split on whitespace; need ≥5 fields. +- If `field[0].lower()` is a known netid (`tcp udp raw sctp dccp mptcp p_raw + p_dgr packet tipc xdp vsock`) **and** ≥6 fields: `state, local, peer = + fields[1], fields[4], fields[5]`; otherwise `fields[0], fields[3], fields[4]`. + (Handles both `ss` layouts — with and without a leading netid column.) +- `inode` from regex `\bino:(\d+)` or `None`. +- `comm`, `pid` from regex `users:\(\("([^"]+)",pid=(\d+),fd=\d+` or `""`/`None`. + +A `None` pid is **meaningful** and must round-trip: it serializes as `?` in +prose and as a distinct key component. `Socket` fields: `state, local, peer, +inode, comm, pid, kind`. + +Derived views: `net_peer_by_inode` (`inode → "local -> peer"`, network sockets +only), `listening_sockets()` (`state == "LISTEN"`), `established_sockets()` +(`state == "ESTAB"`). + +`listener_keys()` maps current listeners to identity keys `port/comm` where +`port = local.rpartition(":")[2]` and `comm` defaults to `?`. This is the +identity used by `new_listener` and by the listener baseline. + +### 1.2.3 SUID scan (expensive, gated) + +`scan_suid_binaries(root="/", extra_dirs=())` (`system.py`): + +1. Walk `root` **staying on its device** (the `find -xdev` equivalent) via an + explicit `os.scandir` stack; collect regular files whose mode has + `S_ISUID | S_ISGID`. +2. **Also** walk each `extra_dirs` entry that is a directory, **ignoring the + device check** — these are the small writable mounts (`/tmp`, `/dev/shm`, …) + that are usually separate tmpfs yet are exactly where a hostile SUID binary + is dropped. +3. Return the **sorted union**. + +Per-entry errors are skipped (fail-open). The walk is expensive, which is why +the daemon gates its cadence (§1.5). + +--- + +## 1.3 Detector registry and gating + +`detectors/__init__.py`. Detectors are pure `detect(state, cfg) -> Iterable +[Alert]`. `REGISTRY` order **is** the order signatures appear in a snapshot and +must be preserved by the port: + +``` +reverse_shell, ld_preload, deleted_exe, input_snooper, credential_access, +stealth_network, memory_obfuscation, egress, +first_seen*, new_listener*, persistence*, new_suid*† +``` + +`*` = `needs_baseline=True`; `†` `new_suid` is also `expensive=True`. + +`run_all(state, cfg, include=None)` yields, per registry entry, when: + +1. `cfg.enabled(name)` is true, and +2. `include` is `None` or contains the name, and +3. if `needs_baseline`: **`state.listener_baseline is not None`**. + +Condition 3 is the arming gate: `listener_baseline is None` skips **all four** +baseline-diff detectors at once — including `first_seen`, even though +`first_seen` never reads `listener_baseline`. `first_seen`'s own gate is this +shared flag, not its store. + +--- + +## 1.4 The daemon sweep loop + +`daemon.py:run()` sequence: + +1. `mkdir log_dir`; write `sentinel.pid`; append a start line to `events.log`. +2. **`build_baselines()`** — see §1.5.1. Note: `run()` *always builds*; it does + not load. +3. `load_fim_baseline()`; `snapshot.prune()`; start eBPF exec + syscall + monitors (each logs enabled/disabled to `events.log`; failure degrades to + `None`, never fatal). +4. Loop until stopped, once per `sample_interval` (default `4.0`s, via + `self._stop.wait(...)` so stop is prompt): + - `write_heartbeat()` (dead-man's switch, every iteration). + - **Only once `(now - start_time) >= baseline_grace`** (default `10`s): kick + the cadence-gated background scanners — SUID, FIM, package verify, pkgdb + anchor, pkgdb signature verify, rootcheck. Each runs off-thread and + self-limits by its own interval; results feed back through + `_on_exec_alert`. + - `alerts = sweep()` (§1.5). + - `fresh = fresh_alerts(alerts, now)` (§1.6). + - If any fresh: capture a snapshot **off the loop thread** (a slow snapshot + must never stall detection; the forensic `SystemState` is rebuilt fresh + inside that thread). + - Every 20 sweeps: `snapshot.prune()`. + +`stop()` sets the event and stops both eBPF monitors. + +Asynchronous producers (eBPF monitors, FIM/pkg/rootcheck threads) do **not** +go through `sweep()`; they call `_on_exec_alert(alert, live_fps=None)`, which +runs the same `fresh_alerts` dedup and then captures off-thread. So the +cooldown and reconciliation chokepoint (§1.6) is shared by loop and async +paths. + +--- + +## 1.5 Baseline lifecycle + +### 1.5.1 Listener and SUID baselines + +Held on the daemon as `set[str]`: `listener_baseline`, `suid_baseline`. + +- **`build_baselines()`** (called by `run()` at start, and by `cli check`): + `listener_baseline = SystemState().listener_keys()` (a *fresh* live capture); + `suid_baseline = set(scan_suid_binaries(extra_dirs=suid_scan_extra_dirs))`. + Both are then written to disk: + - `log_dir/listener-baseline.json` — JSON array, sorted. + - `log_dir/suid-baseline.json` — JSON array, sorted. + + Consequence: **on daemon start the baseline is whatever is running / present + at that moment.** Anything already listening or already SUID at start is, by + definition, baseline and will not alert. The stored files are for the + one-shot CLI path, not reloaded by the running daemon. + +- **`load_baselines()`** reads those two files (`[]` on any error). It is used + **only by `cli check`** (`cli.py`), never by `run()`. + +### 1.5.2 Arming (the grace window) + +Inside `sweep(force_suid=False)`: + +``` +armed = (now - start_time) >= baseline_grace # default 10s +gate = armed or force_suid +state = SystemState( + listener_baseline = self.listener_baseline if gate else None, + suid_baseline = self.suid_baseline, # always passed + suid_binaries = suid_binaries if gate else None, + persist_since = last_persist_scan if gate else None, +) +alerts = list(detectors.run_all(state, cfg)) +last_persist_scan = now # advance AFTER the sweep +``` + +So during the first `baseline_grace` seconds the baseline-diff detectors are +inert (their inputs are `None`), suppressing the startup burst. `suid_baseline` +is always populated but only consumed when `suid_binaries` is present. + +`cli check` passes `force_suid=True`: it bypasses the grace gate **and** +computes `suid_binaries` synchronously via `scan_suid_binaries` (rather than +reading the async result), so a single-shot check gets a complete SUID view. + +### 1.5.3 SUID scan cadence and baseline refresh + +The live SUID list is produced off the loop thread: + +- `_maybe_scan_suid(now)`: skip if a scan thread is alive or + `(now - last_suid_scan) < suid_scan_interval` (default `60`s); else record + `last_suid_scan = now` and spawn `_scan_suid`. +- `_scan_suid()`: `self._suid_current = scan_suid_binaries(...)`. Then, every + `suid_refresh` (default `3600`s), fold the current scan back into the + baseline (`suid_baseline = set(result)`, re-save + `suid-baseline.json`) so legitimately installed SUID binaries stop alerting + after a while. + +`sweep()` reads `self._suid_current`, which is `None` until the first async +scan completes — so `new_suid` is a no-op for the first cycle or two after +start even once armed. A port using synchronous-on-cadence scanning must +reproduce the same observable rule: **no `new_suid` alerts until a scan result +exists**, and periodic baseline refresh on the `suid_refresh` interval. + +### 1.5.4 persist_since + +`last_persist_scan` starts at `start_time` and is set to `now` at the **end** +of every `sweep()`. The `persistence` detector alerts on watched files whose +`mtime > persist_since`, i.e. modified since the previous sweep. Passing the +*previous* sweep's timestamp (advanced afterward) is the contract. + +### 1.5.5 first_seen store + +`detectors/first_seen.py` owns `log_dir/first-seen.json`, independent of the +listener/SUID baselines. Format (schema `enodia.first_seen.v1`): + +```json +{ + "schema": "enodia.first_seen.v1", + "initialized": false, + "public_destinations": { "": ["ip:port", ...] }, + "listener_ports": { "": ["port", ...] } +} +``` + +Behavior each time the detector runs (recall: only when armed): + +1. Load the store (missing/corrupt → the default above). +2. For each `ESTAB` socket to a **public** peer (`netutil.is_public_ip`) with a + port: if `comm`'s `public_destinations` set lacks `ip:port`, add it (list + kept sorted); and **if `initialized`**, emit SID `100074` + `first_public_destination` MEDIUM, key `firstdest::`, + classtype `network-rarity`. +3. For each `LISTEN`/`UNCONN` socket of kind `tcp`/`udp`/`""` with a numeric + non-zero port: if `comm`'s `listener_ports` set lacks it, add it (sorted + numerically); and **if `initialized`**, emit SID `100075` + `first_listener_port` MEDIUM, key `firstlisten::`, classtype + `network-rarity`. +4. Set `initialized = true` and save (atomic: write `*.json.tmp`, `os.replace`, + `chmod 0640` best-effort). + +Net effect: the **first armed run on a fresh store learns silently** +(`initialized` was false) and thereafter alerts once per genuinely new +`comm→destination` / `comm→port`. Because `initialized` persists to disk, a +daemon **restart** does not re-enter the silent-learning phase — unlike the +listener/SUID baselines, which are rebuilt every start. This asymmetry is +deliberate and must be preserved. + +--- + +## 1.6 Dedup, cooldown, and reconciliation chokepoint + +`fresh_alerts(alerts, now, live_fps=None)` is the single gate every alert +passes (loop and async): + +1. **Reconciliation first** — if any alerts, load `ReconcileStore` and + `filter_alerts(alerts, live_fps or {})`, dropping operator-acknowledged + drift so an acked item never even consumes a cooldown slot. `live_fps` + carries current fingerprints for content kinds (e.g. `{"fim": {...}}`); + identity kinds need none. (Detailed in §8.) +2. **Cooldown** — under `_cooldown_lock`, for each surviving alert: emit it and + stamp `cooldowns[key] = now` only if `(now - cooldowns.get(key, 0)) >= + cooldown` (default `60`s). Same `key` inside the window is suppressed. + +Order matters: reconcile before cooldown. + +--- + +## 1.7 Timing knobs (config.py defaults) + +| Key | Default | Governs | +|-----|---------|---------| +| `sample_interval` | `4.0` s | loop period | +| `cooldown` | `60` s | per-key alert dedup window | +| `baseline_grace` | `10` s | arming delay before baseline-diff detectors + background scans | +| `suid_scan_interval` | `60` s | SUID rescan cadence | +| `suid_refresh` | `3600` s | fold current SUID scan into baseline | +| `suid_scan_extra_dirs` | writable mounts tuple | extra (off-device) SUID walk roots | +| `first_seen_enabled` | `true` | first_seen on/off | +| `log_dir` | `/var/log/enodia-sentinel` | store + evidence root; `ENODIA_LOG_DIR` override | + +Store paths derive from `log_dir`: `listener-baseline.json`, +`suid-baseline.json`, `first-seen.json`, `fim-baseline.json`, `events.log`, +`sentinel.pid`. + +--- + +## 1.8 Port parity notes (Go 2.0) + +What the Go agent must reproduce to be behavior-equivalent for this subsystem: + +- **Registry order and gating** (§1.3), including the shared arming gate that + silences `first_seen` while unarmed. +- **Baseline provenance** (§1.5.1): baseline captured *at start*, not loaded, in + the daemon path; loaded in the one-shot path. +- **Time-based arming** (§1.5.2), not first-sweep arming. +- **SUID cadence + refresh + "no alert until a scan exists"** (§1.5.3) — the Go + design uses synchronous-on-cadence scanning, which is allowed as long as the + observable rules hold. +- **`persist_since` = previous sweep time**, advanced after the sweep (§1.5.4). +- **`first-seen.json` format, learn-once semantics, and restart asymmetry** + (§1.5.5); the file format is byte-compatible so either agent can read a store + the other wrote. +- **Reconcile-before-cooldown** ordering and 60s per-key cooldown (§1.6). + +Current Go status (as of this writing): poll detectors are ported and pass the +parity fixture, but baselines are still **injected from fixtures** — the store +package, live SUID walk, and the daemon-equivalent lifecycle above are the +work this document specifies. diff --git a/docs/behavior/index.md b/docs/behavior/index.md new file mode 100644 index 0000000..d5b55c5 --- /dev/null +++ b/docs/behavior/index.md @@ -0,0 +1,51 @@ + +# Enodia Sentinel — Behavioral Specification + +This directory documents the **observable behavior of the current Python +production agent** at reimplementation fidelity: the inputs each subsystem +reads, the algorithm it applies, the outputs it produces (SIDs, severities, +dedup keys, JSON schemas), the files it writes, and its edge-case / fail-open +rules. + +## Why this exists + +Python is the production agent and the **parity oracle** for the Go rewrite +that will ship as Enodia Sentinel **2.0** (see `docs/ROADMAP.md`, Active +Migration Track). "Match the oracle" was previously tribal knowledge encoded +only in the parity fixture and the source itself. These documents make the +contract reviewable: a Go port of any subsystem is correct when it reproduces +the behavior described here. + +This is deliberately distinct from `docs/SPECIFICATION.md`, which defines the +*product* (what Sentinel is, design principles, acceptance criteria). This set +defines *mechanism* (what the code does, precisely enough to re-derive). + +## Conventions + +- **Source anchors** are written as `module.py:function` — the authority is + always the code, and these documents must be updated when the code changes. +- **SIDs / severities / keys** quoted here are the stable contract; a port that + emits a different value for the same condition is a parity break. +- "Fail-open" means a collection or read error degrades to an empty/absent + value and never raises — detection continues on partial data. +- Defaults quoted are the `config.py` dataclass defaults as of package version + `0.7.0`; operators may override them. + +## Sections + +Written in port-priority order. Status reflects which documents exist. + +| # | File | Covers | Status | +|---|------|--------|--------| +| 1 | [`01-sweep-and-state.md`](01-sweep-and-state.md) | `system.py` capture, `daemon.py` sweep loop + baseline lifecycle, `alert.py`, detector registry, `config.py` timing knobs | **written** | +| 2 | `02-poll-detectors.md` | the twelve `detectors/*` — per-detector input → match → output contract | planned | +| 3 | `03-event-detection.md` | `event.py`, exec/syscall/host rule engines, `ruleops.py`, `sids.py` | planned | +| 4 | `04-integrity.md` | `fim.py`, `pkgdb.py`, `selfprotect.py`, `assurance.py` | planned | +| 5 | `05-rootcheck.md` | `rootcheck.py` anti-rootkit cross-view checks | planned | +| 6 | `06-posture.md` | `posture.py` advisory host-hygiene checks | planned | +| 7 | `07-investigation.md` | `snapshot.py`, `incident.py`, `correlation.py`, `enrich.py`, `triage.py` | planned | +| 8 | `08-response.md` | `respond.py` plans, `reconcile.py` acknowledged-drift store | planned | +| 9 | `09-interfaces.md` | `cli.py`, `web.py`, `tui.py` operator-visible contracts | planned | + +Shared helpers (`netutil.py`, `provenance.py`) are documented inline in whatever +section first depends on them.