diff --git a/README.md b/README.md index b7ed81a..1fc47a7 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,9 @@ Project docs: signature, classtype, match fields, expected false positives, and drills. - [Fleet collector design](docs/FLEET_DESIGN.md) — optional v1.2 collector boundaries for small multi-host deployments. +- [Suricata assimilation design](docs/SURICATA_ASSIMILATION.md) — + forward-looking plan for host-scoped Suricata model adoption, Go migration, + sessions, EVE-style events, and triggered forensic packet capture. > **Two implementations, on purpose.** The project began as a bash prototype > (`src/sentinel.sh`, kept as the regression **oracle**) and was re-architected diff --git a/docs/INDEX.md b/docs/INDEX.md index 3c6626c..0ac68e1 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -27,6 +27,7 @@ same patch as any command, schema, or behavior documentation changes. | [ROADMAP.md](ROADMAP.md) | Release tracks and planned sequencing. | | [THREAT_MODEL.md](THREAT_MODEL.md) | Trust boundaries, attacker assumptions, and non-goals. | | [FLEET_DESIGN.md](FLEET_DESIGN.md) | Optional v1.2 collector design: enrollment, ingest, storage, and safety boundaries. | +| [SURICATA_ASSIMILATION.md](SURICATA_ASSIMILATION.md) | Design proposal: assimilate Suricata's models (rule language, sessions, DPI, EVE schema), migrate to Go, add reactive forensic capture. | ## Versioning Rules diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index f2d7922..054bfd4 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -263,6 +263,9 @@ Useful ideas that need design before implementation: - YARA-compatible scanning for selected paths. - Sigma-like host event rules. - SBOM and package provenance reporting. +- Suricata model assimilation: stateful sessions, EVE-style event envelope, + rule metadata/thresholding, datasets, and triggered forensic capture + ([SURICATA_ASSIMILATION.md](SURICATA_ASSIMILATION.md)). - Offline evidence bundle for outside analysis. - Dashboard import of exported incident bundles. - Support for non-pacman package managers in signed-package verification. diff --git a/docs/SURICATA_ASSIMILATION.md b/docs/SURICATA_ASSIMILATION.md new file mode 100644 index 0000000..2e9cb70 --- /dev/null +++ b/docs/SURICATA_ASSIMILATION.md @@ -0,0 +1,287 @@ +# Suricata Assimilation Design + +Status: **Draft / design proposal** +Date: 2026-07-10 +Applies to package line: `0.7.x` (forward-looking; no shipped behavior yet) +Owner doc: sits alongside [SPECIFICATION.md](SPECIFICATION.md) and [ROADMAP.md](ROADMAP.md) + +## 1. Purpose and framing + +This document specifies how Enodia Sentinel should **assimilate the valuable +ideas of [Suricata](https://suricata.io)** — a mature network IDS/IPS/NSM +engine — into an integrated **host security and OS-state monitoring, +administration hub, and framework** that unifies EDR and (host-scoped) IDS/IPS. + +The guiding principle is **idea donor, not engine port**. We take Suricata's +proven *models* (rule language, flow/session correlation, app-layer transaction +parsing, unified event schema, multi-pattern prefiltering, anomaly events, +datasets/reputation) and re-express them in Sentinel's own model — every +signature keeping a stable `sid`, `signature`, and `classtype`, with tests and +operator docs — **without** rebuilding Suricata's line-rate packet data plane. + +It also records the recommended **language migration to Go** and a new +first-class capability: **reactive, triggered forensic packet capture and DPI** +that activates when a cyberattack is detected against the host. + +### 1.1 Target identity (load-bearing assumption) + +Enodia Sentinel is a **host-based** platform: HIDS/HIPS + EDR + OS-state +administration, with **network awareness of the host's own traffic**. It is +**not** a transparent inline network appliance placed between routers doing +always-on multi-gigabit reassembly and verdicting. Every decision below flows +from that assumption; if the identity changes to a line-rate network sensor, +Section 2 (language) and Section 5 (non-goals) must be revisited. + +A true line-rate network appliance (tap/bridge role, inspecting traffic that is +not the host's own) is explicitly deferred to a possible **future sibling +project**, not a mode of Sentinel. It is not a current priority. This keeps +Sentinel's host-scoped identity — and therefore the Go decision — clean, and is +where the Rust escape hatch of Section 2.1 would live if that sibling is ever +built. + +## 2. Language and migration decision + +### 2.1 Recommendation: migrate to Go, keep Rust as an escape hatch + +For an integrated host **hub and framework**, the dominant workload is +orchestration: polling OS state, loading eBPF, correlating events, serving a +console, and performing reviewed administration actions. That is Go's sweet +spot. Decisive factors: + +- **eBPF without a runtime toolchain.** Today the event layer shells to `bcc`, + which drags Python + clang into the *runtime*. Go's + [`cilium/ebpf`](https://github.com/cilium/ebpf) is a pure-Go CO-RE loader: + one static binary, no bcc, no clang. This removes Sentinel's largest + deployment dependency and matches the "useful without extras, easy to deploy" + ethos. +- **Concurrency model.** Goroutines map cleanly onto per-flow, per-detector, + and per-connection work. +- **Framework surface.** A plugin/extension framework wants stable interfaces + and a single auditable binary — both natural in Go. +- **Velocity.** A broad-scope platform needs fast iteration and a low + contributor barrier. + +**Rust would only win** if Sentinel committed to line-rate inline *network* +packet inspection — which the identity statement explicitly rejects. Suricata +adopted Rust for its untrusted-packet data plane; Sentinel has no such data +plane and therefore does not inherit that requirement. If a hot untrusted-input +parsing core ever appears (e.g. a bespoke protocol decoder proves to be a +bottleneck), introduce a **small Rust component behind a stable interface** +then — do not pay the whole-rewrite tax now. + +### 2.2 The dependency-discipline caveat + +Sentinel's Python core is *stdlib-only*. A Go rewrite cannot be literally +zero-dependency: practical eBPF and packet capture require vetted libraries +(`cilium/ebpf`, `gopacket`). The ethos is preserved as **"few, audited, +security-relevant dependencies, one static binary"** rather than pure stdlib. +This is an explicit, documented tradeoff, not a drift. The console +(`net/http` + `crypto/tls`), JSON (`encoding/json`), and config remain stdlib. + +### 2.3 Migration strategy: strangler via stable schemas + +The Python implementation is not thrown away; it becomes the **behavioral +oracle** during transition, exactly as the bash prototype (`src/sentinel.sh`) +is the oracle for the Python port today. + +- **Schemas are the contract.** The v1 JSON schemas in + [SCHEMAS.md](SCHEMAS.md) (`enodia.alert.v1`, `enodia.incident.v1`, + `enodia.status.v1`, …) define the interface between Python and Go. The Go + agent must emit byte-comparable structured output for the same inputs. +- **One red-team harness, three implementations.** The existing red-team drill + harness runs against bash, Python, and (incrementally) Go. A subsystem is + "cut over" only when Go matches the oracle on every relevant `sid`. +- **Subsystem-by-subsystem.** Ported in the phase order of Section 6; Python + keeps ownership of anything not yet at parity. No big-bang switch. + +## 3. Suricata feature-assimilation catalog + +Each row is a Suricata idea, its value, and how it maps into Sentinel's model. +"Keep" = adopt; "adapt" = adopt with host-scoping; "skip" = out of scope +(Section 5). + +| Suricata concept | Value | Sentinel mapping | Verdict | +|---|---|---|---| +| Rule metadata: `sid`, `rev`, `msg`, `classtype`, `priority`, `reference`, `metadata` | Stable, referenceable, tunable signatures | Sentinel already has `sid`/`signature`/`classtype`/`severity`; add `rev`, `reference`, `metadata` to the rule model in `ruleops.py` | Keep | +| `flowbits` (set/isset/unset/toggle/noalert) | Stateful correlation across events in one flow | A **"statebits"** facility on the session object (Section 3.1) so a rule can fire only after a prior condition in the same session | Adapt | +| `threshold` / `detection_filter` (limit, threshold, both; by src/dst) | Rate-limits noisy signatures | Per-`sid` thresholding in the rule engine; mirror Suricata semantics (limit/threshold/both, per-scope, per-window) | Keep | +| `suppress` | Silences known-benign matches by scope | Extends the existing `reconcile.py` acknowledged-drift model to rule-level suppression by `sid` + scope | Adapt | +| Fast-pattern / **MPM** (Aho-Corasick) prefilter | Scales to thousands of content rules cheaply | An Aho-Corasick prefilter stage in front of full rule evaluation, over argv/paths/captured payloads — introduced only when rule count justifies it (YAGNI gate) | Adapt | +| **Flow engine** (bidirectional flow hash, state, timeouts) | The correlation backbone | A host-scoped **session/flow model** (Section 3.1) keyed on 5-tuple *and* owning PID lineage | Adapt | +| **App-layer transactions** + protocol detection | Structured per-request/response records | Applied only to *triggered* captures (Section 4), producing per-transaction records for enrichment | Adapt | +| **EVE** unified JSON (`event_type`: alert, flow, dns, tls, http, quic, ssh, fileinfo, anomaly) | One machine-readable envelope | Formalize `schemas.py` into an **EVE-style event envelope** with a typed `event_type` discriminator; existing alert/incident schemas become event types | Keep | +| **JA3/JA4** TLS fingerprinting | Identify clients/malware from TLS ClientHello | JA4 computation over triggered TLS captures, attached to the incident | Keep | +| Decoder / **anomaly** events | "Malformed input is itself a signal" | Anomaly events from capture DPI (truncated/invalid app-layer) as first-class low-severity signals | Adapt | +| **Datasets / datajson / iprep** (bulk IOC + reputation) | Match against large IP/domain/hash sets | A dataset store feeding `egress`, DNS, and TLS signals; explicit IOC lists distinct from behavioral heuristics | Adapt | +| **Filestore** + file hashing | Extract and hash transferred files | Bounded extraction from triggered captures, hashed and attached as evidence; reuse `provenance.py` hashing conventions | Adapt | +| Stream (TCP) reassembly | Ordered bytes for app-layer parsing | Only *within a captured flow* for DPI — not an always-on reassembly engine | Adapt | + +### 3.1 The session/flow correlation model + +Sentinel today is process/host-centric; Suricata is flow-centric. The +assimilation is a **session object** that unifies both and realizes the +roadmap's "correlation across exec, network, persistence, FIM, and rootcheck": + +- **Key:** owning process lineage **and** any associated network 5-tuples. +- **Aggregates:** exec events, syscall events, network connections, FIM/pkg + changes, rootcheck findings, and posture context observed for that lineage + within a time window. +- **Carries statebits:** the `flowbits`-equivalent, enabling rules like + "credential file opened *then* egress to a public IP in the same session." +- **Feeds incidents:** `incident.py`'s current lineage+window grouping becomes + a projection over sessions; snapshots attach session evidence. + +## 4. Reactive forensic capture & DPI + +A packet inspector and sniffer that **activates when an attack is detected**, +preserving wire evidence the way Sentinel already preserves process/disk +evidence. It is bounded and event-driven — not always-on inspection — which is +why it does not change the Go language call. + +### 4.1 Trigger model + +- High-severity detections (`reverse_shell`, `egress`/C2, `new_listener`, + `stealth_network`, C2 exec rules, unusual interpreter bind/accept/egress + SIDs) mark their associated 5-tuple / PID as a **flow of interest** and fire + a capture trigger. +- Triggers are debounced per session so one incident does not spawn redundant + captures. + +### 4.2 Pre-trigger ring buffer (approved) + +The interesting packets precede the alert. Sentinel maintains a **small, +in-memory rolling capture** *only for flows already flagged as of-interest* +(interpreter-owned sockets, new listeners, egress to public IPs) — never every +packet on the host. On trigger, the ring buffer is flushed to evidence so the +capture includes the **preceding context**, then capture continues for a +bounded window. Buffer is byte- and time-capped per flow; oldest data is +overwritten. + +### 4.3 Tight scoping and hard caps (approved, mandatory) + +- **Scope:** capture is restricted to the offending 5-tuple / peer IP / owning + PID. No host-wide capture. +- **Caps:** per-capture time window, per-capture byte ceiling, global + max-concurrent-captures, and a global disk-retention ceiling. A hostile flow + must not be able to turn Sentinel's own sniffer into a disk-fill or + memory-exhaustion DoS. +- **Fail-safe:** if `CAP_NET_RAW`/libpcap/AF_PACKET is unavailable, log the + reason and degrade to **metadata-only** evidence (5-tuples and process + linkage from `/proc` + eBPF). Capture failure must never disable detection — + mirroring the existing bcc/eBPF fail-safe posture. + +### 4.4 DPI enrichment (assimilated from Suricata) + +Within the captured, scoped flow only, Sentinel reassembles and runs app-layer +heuristics to enrich the incident: **TLS SNI + JA4**, **DNS** queries/answers, +**HTTP** host/URI, **QUIC**, and generic protocol/anomaly detection. Output is +**EVE-style event records** (Section 3, `event_type` = `tls`/`dns`/`http`/…) +attached to the incident. + +### 4.5 Evidence and audit + +- Writes a scoped `.pcap` plus parsed EVE-style events under the incident's + evidence directory; attaches them to the snapshot and incident. +- Appends a **capture audit entry** (who/what/when/scope/caps) analogous to + `response-audit.log`. +- Capture events carry their own `sid` range and `classtype` + (`evidence.capture`). + +### 4.6 Privacy / threat-model gating (approved) + +Payload capture is sensitive (can contain credentials, PII). It remains +evidence-preservation (read-only observation, consistent with the +explicit-not-silent stance) but is gated: + +- **Opt-in config** (`capture.enabled`), default off. +- **Payload tier gating:** headers/metadata-only vs. full-payload as separate + config levels. +- **Retention TTL** on captured artifacts; audited deletion. +- **Documented** in [THREAT_MODEL.md](THREAT_MODEL.md): what is captured, the + privacy exposure, and the operator's legal/consent responsibility. +- **No silent enable**, no capture outside configured interface/scope. + +## 5. What NOT to assimilate (YAGNI / non-goals) + +- **No line-rate inline data plane.** No always-on full-traffic DPI, no packet + verdicting/dropping at multi-gigabit. +- **No high-speed capture backends.** No DPDK, AF_XDP, netmap, PF_RING, + Napatech, or RSS/fanout scaling. Triggered libpcap/AF_PACKET is sufficient. +- **No transparent bridge/tap deployment.** Sentinel watches the host's own + traffic, not a span port between other devices. +- **No C ABI / embeddable-library surface** (`libsuricata` analogue). +- **No distributed sensor mesh here.** Fleet/collector work stays in its own + design (`FLEET_DESIGN.md`), not this document. +- **No stealth, self-hiding, or automatic destructive remediation** — unchanged + from Sentinel's standing rules. + +## 6. Phased roadmap + +Each phase preserves the oracle discipline: bash + Python remain authoritative +until Go reaches per-subsystem parity against the red-team harness, and every +signature keeps `sid` / `signature` / `classtype` / tests / docs. + +- **Phase 0 — Schema-first.** Freeze and extend the v1 EVE-style event envelope + in [SCHEMAS.md](SCHEMAS.md) (`event_type` discriminator; alert/incident/status + as event types). This is the Python↔Go contract. *No language change yet.* +- **Phase 1 — Go agent skeleton.** Config, daemon sweep loop, `SystemState` + equivalent, poll detectors ported, JSON output matching schemas. Runs + alongside Python; harness compares both. +- **Phase 2 — eBPF via `cilium/ebpf`.** Port exec/syscall rules; drop the + bcc/Python runtime dependency; single static binary. +- **Phase 3 — Rule-engine parity + assimilation.** `rev`/`reference`/`metadata`, + statebits (flowbits), thresholding, suppression, and — if rule count warrants + — the MPM prefilter. `rules list/show/test/docs` parity. +- **Phase 4 — Session/correlation model.** Unify exec + network + FIM + + rootcheck into sessions; re-project incidents over sessions. +- **Phase 5 — Network awareness + reactive forensic capture & DPI.** Ring + buffer, triggered scoped capture, app-layer enrichment (JA4/DNS/TLS/HTTP/QUIC), + datasets/reputation, evidence + audit + privacy gating. +- **Phase 6 — Framework/plugin surface.** Go interfaces for detectors, rules, + response actions, and notify backends; administration-hub features; console + parity. + +## 7. Constraints preserved + +- Detection stays useful without a dashboard, collector, or network. +- Read-only detection and dry-run/reviewed response until schemas and audit + logs are tested and documented; **no automatic inline blocking** without an + explicit, reviewed, audited `--apply` path. +- Every new detector, rule, response action, or capture event needs a stable + `sid`, `signature`, `classtype`, tests, and operator docs. +- No stealth, self-hiding, persistence tricks, or automatic destructive + remediation. +- The bash prototype and red-team harness remain the behavioral oracle through + the migration. + +## 8. Acceptance criteria + +1. A written, reviewed schema (Phase 0) that both Python and a Go prototype can + emit and that passes existing compatibility tests. +2. For each ported subsystem, Go output matches the Python/bash oracle on every + relevant `sid` in the red-team harness before cutover. +3. Triggered capture demonstrably: (a) includes pre-trigger context from the + ring buffer, (b) never exceeds configured time/byte/concurrency/retention + caps under a flood, (c) degrades to metadata-only when capture privileges are + absent, (d) is disabled by default and audited when enabled. +4. Every assimilated rule feature (statebits, thresholding, suppression) has + fixture/red-team coverage and generated `RULES.md` documentation. + +## 9. Open questions + +- Go eBPF: `cilium/ebpf` (CO-RE, recommended) vs. a thin cgo shim to existing + probes during transition? +- Capture library: `gopacket` + AF_PACKET vs. an eBPF/`AF_PACKET` custom path + for tighter cap enforcement? +- Does the session model supersede or wrap the current `incident.py` grouping, + and what is the migration for persisted incidents? +- Dataset/reputation feed sourcing and update policy (offline-first, signed). + +## 10. Documentation impact (when work begins) + +Per Sentinel's doc checklist, phased work must update: `SPECIFICATION.md` +(product/data model), `ROADMAP.md` (sequencing), `SCHEMAS.md` (event envelope), +`THREAT_MODEL.md` (capture privacy boundary), `COMMAND_REFERENCE.md` (capture + +rule commands), `RULES.md` (new SIDs), `RUNBOOKS.md` (capture-evidence +workflow), `config/enodia-sentinel.toml` (`capture.*` keys), and `INDEX.md` + +`VERSION.json` (versioning).