luna/enodia-sentinal
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
enodia-sentinal/docs/behavior/01-sweep-and-state.md

15 KiB
Raw Blame History

1. Sweep, System State, and Baseline Lifecycle

Scope: how the daemon gathers one view of the host, runs poll detectors over it, and manages the baselines that the baseline-diff detectors (first_seen, new_listener, new_suid, persistence) depend on. This is the substrate every other detection subsystem runs on top of.

Source: system.py, daemon.py, alert.py, detectors/__init__.py, config.py.

1.1 The alert value

alert.py. Every detection is an immutable Alert:

Field Type Meaning
severity Severity enum MEDIUM=1, HIGH=2, CRITICAL=3; str() yields the name
signature str human rule name, e.g. new_listener
key str stable dedup identity, e.g. lis:31337/nc; drives cooldown
detail str one-line human description
pids tuple[int, …] processes the snapshot should deep-dive (default empty)
sid int stable Snort-style signature id (default 0)
classtype str category (default uncategorized)

Alert.to_dict() is the serialization contract — key order: sid, severity, signature, classtype, key, detail, pids (with severity as the string name and pids as a list). A port must preserve key values exactly: cooldown dedup is keyed on them, so a differing key changes re-fire behavior.

1.2 System state capture

SystemState (system.py) is one sweep's cached view, built once and shared by all detectors. It is fully injectable: constructing it with processes=/sockets= bypasses live collection (how tests and the parity harness avoid /proc and ss).

1.2.1 Processes

SystemState.processes enumerates /proc, keeping entries whose name isdigit(), each wrapped in a Process(pid). Every Process field is a cached_property read lazily from /proc/<pid>/… and fail-open (any OSError → empty value), because a process can exit mid-sweep:

Field Source Notes
comm comm stripped
cmdline cmdline NUL→space, stripped
exe readlink exe "" if unreadable
cwd readlink cwd
environ environ NUL-split k=v → dict
fd_targets fd/ dict fd(str) → link target, fds sorted numerically
memory_maps maps see parser below
status status key → value (value stripped)
ppid status[PPid] 0 on parse failure
uid status[Uid] field 0 0 if absent

stdio_socket_inode() returns the socket inode if fd 0, 1, or 2 links to socket:[…] — the reverse-shell tell — else None.

parse_memory_maps(text) splits each line maxsplit=5; needs ≥5 fields; address start-end parsed as hex (skip line if no - or non-hex); path is field 5 when present else "". Emits MemoryMap(start, end, perms, path). The port must preserve maxsplit=5 semantics so pathnames containing spaces stay intact.

1.2.2 Sockets

SystemState.sockets runs ss -H once per protocol family, in this order, tagging each result with kind:

-tanep tcp   -uanep udp   -wanep raw   -Sanep sctp   -danep dccp
-0anep packet   -Manep mptcp   --tipc -anep tipc   --xdp -anep xdp
--vsock -anep vsock

Each ss call is fail-open (OSError/SubprocessError/10s timeout → ""), so an unsupported family drops only its own rows. _parse_ss per line:

  • Split on whitespace; need ≥5 fields.
  • If field[0].lower() is a known netid (tcp udp raw sctp dccp mptcp p_raw p_dgr packet tipc xdp vsock) and ≥6 fields: state, local, peer = fields[1], fields[4], fields[5]; otherwise fields[0], fields[3], fields[4]. (Handles both ss layouts — with and without a leading netid column.)
  • inode from regex \bino:(\d+) or None.
  • comm, pid from regex users:\(\("([^"]+)",pid=(\d+),fd=\d+ or ""/None.

A None pid is meaningful and must round-trip: it serializes as ? in prose and as a distinct key component. Socket fields: state, local, peer, inode, comm, pid, kind.

Derived views: net_peer_by_inode (inode → "local -> peer", network sockets only), listening_sockets() (state == "LISTEN"), established_sockets() (state == "ESTAB").

listener_keys() maps current listeners to identity keys port/comm where port = local.rpartition(":")[2] and comm defaults to ?. This is the identity used by new_listener and by the listener baseline.

1.2.3 SUID scan (expensive, gated)

scan_suid_binaries(root="/", extra_dirs=()) (system.py):

  1. Walk root staying on its device (the find -xdev equivalent) via an explicit os.scandir stack; collect regular files whose mode has S_ISUID | S_ISGID.
  2. Also walk each extra_dirs entry that is a directory, ignoring the device check — these are the small writable mounts (/tmp, /dev/shm, …) that are usually separate tmpfs yet are exactly where a hostile SUID binary is dropped.
  3. Return the sorted union.

Per-entry errors are skipped (fail-open). The walk is expensive, which is why the daemon gates its cadence (§1.5).

1.3 Detector registry and gating

detectors/__init__.py. Detectors are pure detect(state, cfg) -> Iterable [Alert]. REGISTRY order is the order signatures appear in a snapshot and must be preserved by the port:

reverse_shell, ld_preload, deleted_exe, input_snooper, credential_access,
stealth_network, memory_obfuscation, egress,
first_seen*, new_listener*, persistence*, new_suid*†

* = needs_baseline=True; † new_suid is also expensive=True.

run_all(state, cfg, include=None) yields, per registry entry, when:

  1. cfg.enabled(name) is true, and
  2. include is None or contains the name, and
  3. if needs_baseline: state.listener_baseline is not None.

Condition 3 is the arming gate: listener_baseline is None skips all four baseline-diff detectors at once — including first_seen, even though first_seen never reads listener_baseline. first_seen's own gate is this shared flag, not its store.

1.4 The daemon sweep loop

daemon.py:run() sequence:

  1. mkdir log_dir; write sentinel.pid; append a start line to events.log.
  2. build_baselines() — see §1.5.1. Note: run() always builds; it does not load.
  3. load_fim_baseline(); snapshot.prune(); start eBPF exec + syscall monitors (each logs enabled/disabled to events.log; failure degrades to None, never fatal).
  4. Loop until stopped, once per sample_interval (default 4.0s, via self._stop.wait(...) so stop is prompt):
    • write_heartbeat() (dead-man's switch, every iteration).
    • Only once (now - start_time) >= baseline_grace (default 10s): kick the cadence-gated background scanners — SUID, FIM, package verify, pkgdb anchor, pkgdb signature verify, rootcheck. Each runs off-thread and self-limits by its own interval; results feed back through _on_exec_alert.
    • alerts = sweep() (§1.5).
    • fresh = fresh_alerts(alerts, now) (§1.6).
    • If any fresh: capture a snapshot off the loop thread (a slow snapshot must never stall detection; the forensic SystemState is rebuilt fresh inside that thread).
    • Every 20 sweeps: snapshot.prune().

stop() sets the event and stops both eBPF monitors.

Asynchronous producers (eBPF monitors, FIM/pkg/rootcheck threads) do not go through sweep(); they call _on_exec_alert(alert, live_fps=None), which runs the same fresh_alerts dedup and then captures off-thread. So the cooldown and reconciliation chokepoint (§1.6) is shared by loop and async paths.

1.5 Baseline lifecycle

1.5.1 Listener and SUID baselines

Held on the daemon as set[str]: listener_baseline, suid_baseline.

  • build_baselines() (called by run() at start, and by cli check): listener_baseline = SystemState().listener_keys() (a fresh live capture); suid_baseline = set(scan_suid_binaries(extra_dirs=suid_scan_extra_dirs)). Both are then written to disk:

    • log_dir/listener-baseline.json — JSON array, sorted.
    • log_dir/suid-baseline.json — JSON array, sorted.

    Consequence: on daemon start the baseline is whatever is running / present at that moment. Anything already listening or already SUID at start is, by definition, baseline and will not alert. The stored files are for the one-shot CLI path, not reloaded by the running daemon.

  • load_baselines() reads those two files ([] on any error). It is used only by cli check (cli.py), never by run().

1.5.2 Arming (the grace window)

Inside sweep(force_suid=False):

armed = (now - start_time) >= baseline_grace         # default 10s
gate  = armed or force_suid
state = SystemState(
    listener_baseline = self.listener_baseline if gate else None,
    suid_baseline     = self.suid_baseline,          # always passed
    suid_binaries     = suid_binaries if gate else None,
    persist_since     = last_persist_scan if gate else None,
)
alerts = list(detectors.run_all(state, cfg))
last_persist_scan = now                              # advance AFTER the sweep

So during the first baseline_grace seconds the baseline-diff detectors are inert (their inputs are None), suppressing the startup burst. suid_baseline is always populated but only consumed when suid_binaries is present.

cli check passes force_suid=True: it bypasses the grace gate and computes suid_binaries synchronously via scan_suid_binaries (rather than reading the async result), so a single-shot check gets a complete SUID view.

1.5.3 SUID scan cadence and baseline refresh

The live SUID list is produced off the loop thread:

  • _maybe_scan_suid(now): skip if a scan thread is alive or (now - last_suid_scan) < suid_scan_interval (default 60s); else record last_suid_scan = now and spawn _scan_suid.
  • _scan_suid(): self._suid_current = scan_suid_binaries(...). Then, every suid_refresh (default 3600s), fold the current scan back into the baseline (suid_baseline = set(result), re-save suid-baseline.json) so legitimately installed SUID binaries stop alerting after a while.

sweep() reads self._suid_current, which is None until the first async scan completes — so new_suid is a no-op for the first cycle or two after start even once armed. A port using synchronous-on-cadence scanning must reproduce the same observable rule: no new_suid alerts until a scan result exists, and periodic baseline refresh on the suid_refresh interval.

1.5.4 persist_since

last_persist_scan starts at start_time and is set to now at the end of every sweep(). The persistence detector alerts on watched files whose mtime > persist_since, i.e. modified since the previous sweep. Passing the previous sweep's timestamp (advanced afterward) is the contract.

1.5.5 first_seen store

detectors/first_seen.py owns log_dir/first-seen.json, independent of the listener/SUID baselines. Format (schema enodia.first_seen.v1):

{
  "schema": "enodia.first_seen.v1",
  "initialized": false,
  "public_destinations": { "<comm>": ["ip:port", ...] },
  "listener_ports":      { "<comm>": ["port", ...] }
}

Behavior each time the detector runs (recall: only when armed):

  1. Load the store (missing/corrupt → the default above).
  2. For each ESTAB socket to a public peer (netutil.is_public_ip) with a port: if comm's public_destinations set lacks ip:port, add it (list kept sorted); and if initialized, emit SID 100074 first_public_destination MEDIUM, key firstdest:<comm>:<ip:port>, classtype network-rarity.
  3. For each LISTEN/UNCONN socket of kind tcp/udp/"" with a numeric non-zero port: if comm's listener_ports set lacks it, add it (sorted numerically); and if initialized, emit SID 100075 first_listener_port MEDIUM, key firstlisten:<comm>:<port>, classtype network-rarity.
  4. Set initialized = true and save (atomic: write *.json.tmp, os.replace, chmod 0640 best-effort).

Net effect: the first armed run on a fresh store learns silently (initialized was false) and thereafter alerts once per genuinely new comm→destination / comm→port. Because initialized persists to disk, a daemon restart does not re-enter the silent-learning phase — unlike the listener/SUID baselines, which are rebuilt every start. This asymmetry is deliberate and must be preserved.

1.6 Dedup, cooldown, and reconciliation chokepoint

fresh_alerts(alerts, now, live_fps=None) is the single gate every alert passes (loop and async):

  1. Reconciliation first — if any alerts, load ReconcileStore and filter_alerts(alerts, live_fps or {}), dropping operator-acknowledged drift so an acked item never even consumes a cooldown slot. live_fps carries current fingerprints for content kinds (e.g. {"fim": {...}}); identity kinds need none. (Detailed in §8.)
  2. Cooldown — under _cooldown_lock, for each surviving alert: emit it and stamp cooldowns[key] = now only if (now - cooldowns.get(key, 0)) >= cooldown (default 60s). Same key inside the window is suppressed.

Order matters: reconcile before cooldown.

1.7 Timing knobs (config.py defaults)

Key Default Governs
sample_interval 4.0 s loop period
cooldown 60 s per-key alert dedup window
baseline_grace 10 s arming delay before baseline-diff detectors + background scans
suid_scan_interval 60 s SUID rescan cadence
suid_refresh 3600 s fold current SUID scan into baseline
suid_scan_extra_dirs writable mounts tuple extra (off-device) SUID walk roots
first_seen_enabled true first_seen on/off
log_dir /var/log/enodia-sentinel store + evidence root; ENODIA_LOG_DIR override

Store paths derive from log_dir: listener-baseline.json, suid-baseline.json, first-seen.json, fim-baseline.json, events.log, sentinel.pid.

1.8 Port parity notes (Go port)

What the Go agent must reproduce to be behavior-equivalent for this subsystem:

  • Registry order and gating (§1.3), including the shared arming gate that silences first_seen while unarmed.
  • Baseline provenance (§1.5.1): baseline captured at start, not loaded, in the daemon path; loaded in the one-shot path.
  • Time-based arming (§1.5.2), not first-sweep arming.
  • SUID cadence + refresh + "no alert until a scan exists" (§1.5.3) — the Go design uses synchronous-on-cadence scanning, which is allowed as long as the observable rules hold.
  • persist_since = previous sweep time, advanced after the sweep (§1.5.4).
  • first-seen.json format, learn-once semantics, and restart asymmetry (§1.5.5); the file format is byte-compatible so either agent can read a store the other wrote.
  • Reconcile-before-cooldown ordering and 60s per-key cooldown (§1.6).

Current Go status: poll detectors pass the parity fixture. The baseline store, live SUID walk, startup grace, persistence scan, and first-seen retention are implemented behind the sidecar's explicit --state-dir lifecycle. Fixture mode continues to inject state and never touches a live store. Production service cutover remains outside this acceptance slice.