enodia-sentinal/docs/SURICATA_ASSIMILATION.md

# Suricata Assimilation Design

Status: **Draft / design proposal**
Date: 2026-07-10
Applies to package line: `0.7.x` (forward-looking; no shipped behavior yet)
Owner doc: sits alongside [SPECIFICATION.md](SPECIFICATION.md) and [ROADMAP.md](ROADMAP.md)

## 1. Purpose and framing

This document specifies how Enodia Sentinel should **assimilate the valuable
ideas of [Suricata](https://suricata.io)** — a mature network IDS/IPS/NSM
engine — into an integrated **host security and OS-state monitoring,
administration hub, and framework** that unifies EDR and (host-scoped) IDS/IPS.

The guiding principle is **idea donor, not engine port**. We take Suricata's
proven *models* (rule language, flow/session correlation, app-layer transaction
parsing, unified event schema, multi-pattern prefiltering, anomaly events,
datasets/reputation) and re-express them in Sentinel's own model — every
signature keeping a stable `sid`, `signature`, and `classtype`, with tests and
operator docs — **without** rebuilding Suricata's line-rate packet data plane.

It also records the recommended **language migration to Go** and a new
first-class capability: **reactive, triggered forensic packet capture and DPI**
that activates when a cyberattack is detected against the host.

### 1.1 Target identity (load-bearing assumption)

Enodia Sentinel is a **host-based** platform: HIDS/HIPS + EDR + OS-state
administration, with **network awareness of the host's own traffic**. It is
**not** a transparent inline network appliance placed between routers doing
always-on multi-gigabit reassembly and verdicting. Every decision below flows
from that assumption; if the identity changes to a line-rate network sensor,
Section 2 (language) and Section 5 (non-goals) must be revisited.

A true line-rate network appliance (tap/bridge role, inspecting traffic that is
not the host's own) is explicitly deferred to a possible **future sibling
project**, not a mode of Sentinel. It is not a current priority. This keeps
Sentinel's host-scoped identity — and therefore the Go decision — clean, and is
where the Rust escape hatch of Section 2.1 would live if that sibling is ever
built.

## 2. Language and migration decision

### 2.1 Recommendation: migrate to Go, keep Rust as an escape hatch

For an integrated host **hub and framework**, the dominant workload is
orchestration: polling OS state, loading eBPF, correlating events, serving a
console, and performing reviewed administration actions. That is Go's sweet
spot. Decisive factors:

- **eBPF without a runtime toolchain.** Today the event layer shells to `bcc`,
  which drags Python + clang into the *runtime*. Go's
  [`cilium/ebpf`](https://github.com/cilium/ebpf) is a pure-Go CO-RE loader:
  one static binary, no bcc, no clang. This removes Sentinel's largest
  deployment dependency and matches the "useful without extras, easy to deploy"
  ethos.
- **Concurrency model.** Goroutines map cleanly onto per-flow, per-detector,
  and per-connection work.
- **Framework surface.** A plugin/extension framework wants stable interfaces
  and a single auditable binary — both natural in Go.
- **Velocity.** A broad-scope platform needs fast iteration and a low
  contributor barrier.

**Rust would only win** if Sentinel committed to line-rate inline *network*
packet inspection — which the identity statement explicitly rejects. Suricata
adopted Rust for its untrusted-packet data plane; Sentinel has no such data
plane and therefore does not inherit that requirement. If a hot untrusted-input
parsing core ever appears (e.g. a bespoke protocol decoder proves to be a
bottleneck), introduce a **small Rust component behind a stable interface**
then — do not pay the whole-rewrite tax now.

### 2.2 The dependency-discipline caveat

Sentinel's Python core is *stdlib-only*. A Go rewrite cannot be literally
zero-dependency: practical eBPF and packet capture require vetted libraries
(`cilium/ebpf`, `gopacket`). The ethos is preserved as **"few, audited,
security-relevant dependencies, one static binary"** rather than pure stdlib.
This is an explicit, documented tradeoff, not a drift. The console
(`net/http` + `crypto/tls`), JSON (`encoding/json`), and config remain stdlib.

### 2.3 Migration strategy: strangler via stable schemas

The Python implementation is not thrown away; it becomes the **behavioral
oracle** during transition, exactly as the bash prototype (`src/sentinel.sh`)
is the oracle for the Python port today.

- **Schemas are the contract.** The v1 JSON schemas in
  [SCHEMAS.md](SCHEMAS.md) (`enodia.alert.v1`, `enodia.incident.v1`,
  `enodia.status.v1`, …) define the interface between Python and Go. The Go
  agent must emit byte-comparable structured output for the same inputs.
- **One red-team harness, three implementations.** The existing red-team drill
  harness runs against bash, Python, and (incrementally) Go. A subsystem is
  "cut over" only when Go matches the oracle on every relevant `sid`.
- **Subsystem-by-subsystem.** Ported in the phase order of Section 6; Python
  keeps ownership of anything not yet at parity. No big-bang switch.

## 3. Suricata feature-assimilation catalog

Each row is a Suricata idea, its value, and how it maps into Sentinel's model.
"Keep" = adopt; "adapt" = adopt with host-scoping; "skip" = out of scope
(Section 5).

| Suricata concept | Value | Sentinel mapping | Verdict |
|---|---|---|---|
| Rule metadata: `sid`, `rev`, `msg`, `classtype`, `priority`, `reference`, `metadata` | Stable, referenceable, tunable signatures | Sentinel already has `sid`/`signature`/`classtype`/`severity`; add `rev`, `reference`, `metadata` to the rule model in `ruleops.py` | Keep |
| `flowbits` (set/isset/unset/toggle/noalert) | Stateful correlation across events in one flow | A **"statebits"** facility on the session object (Section 3.1) so a rule can fire only after a prior condition in the same session | Adapt |
| `threshold` / `detection_filter` (limit, threshold, both; by src/dst) | Rate-limits noisy signatures | Per-`sid` thresholding in the rule engine; mirror Suricata semantics (limit/threshold/both, per-scope, per-window) | Keep |
| `suppress` | Silences known-benign matches by scope | Extends the existing `reconcile.py` acknowledged-drift model to rule-level suppression by `sid` + scope | Adapt |
| Fast-pattern / **MPM** (Aho-Corasick) prefilter | Scales to thousands of content rules cheaply | An Aho-Corasick prefilter stage in front of full rule evaluation, over argv/paths/captured payloads — introduced only when rule count justifies it (YAGNI gate) | Adapt |
| **Flow engine** (bidirectional flow hash, state, timeouts) | The correlation backbone | A host-scoped **session/flow model** (Section 3.1) keyed on 5-tuple *and* owning PID lineage | Adapt |
| **App-layer transactions** + protocol detection | Structured per-request/response records | Applied only to *triggered* captures (Section 4), producing per-transaction records for enrichment | Adapt |
| **EVE** unified JSON (`event_type`: alert, flow, dns, tls, http, quic, ssh, fileinfo, anomaly) | One machine-readable envelope | Formalize `schemas.py` into an **EVE-style event envelope** with a typed `event_type` discriminator; existing alert/incident schemas become event types | Keep |
| **JA3/JA4** TLS fingerprinting | Identify clients/malware from TLS ClientHello | JA4 computation over triggered TLS captures, attached to the incident | Keep |
| Decoder / **anomaly** events | "Malformed input is itself a signal" | Anomaly events from capture DPI (truncated/invalid app-layer) as first-class low-severity signals | Adapt |
| **Datasets / datajson / iprep** (bulk IOC + reputation) | Match against large IP/domain/hash sets | A dataset store feeding `egress`, DNS, and TLS signals; explicit IOC lists distinct from behavioral heuristics | Adapt |
| **Filestore** + file hashing | Extract and hash transferred files | Bounded extraction from triggered captures, hashed and attached as evidence; reuse `provenance.py` hashing conventions | Adapt |
| Stream (TCP) reassembly | Ordered bytes for app-layer parsing | Only *within a captured flow* for DPI — not an always-on reassembly engine | Adapt |

### 3.1 The session/flow correlation model

Sentinel today is process/host-centric; Suricata is flow-centric. The
assimilation is a **session object** that unifies both and realizes the
roadmap's "correlation across exec, network, persistence, FIM, and rootcheck":

- **Key:** owning process lineage **and** any associated network 5-tuples.
- **Aggregates:** exec events, syscall events, network connections, FIM/pkg
  changes, rootcheck findings, and posture context observed for that lineage
  within a time window.
- **Carries statebits:** the `flowbits`-equivalent, enabling rules like
  "credential file opened *then* egress to a public IP in the same session."
- **Feeds incidents:** `incident.py`'s current lineage+window grouping becomes
  a projection over sessions; snapshots attach session evidence.

## 4. Reactive forensic capture & DPI

A packet inspector and sniffer that **activates when an attack is detected**,
preserving wire evidence the way Sentinel already preserves process/disk
evidence. It is bounded and event-driven — not always-on inspection — which is
why it does not change the Go language call.

### 4.1 Trigger model

- High-severity detections (`reverse_shell`, `egress`/C2, `new_listener`,
  `stealth_network`, C2 exec rules, unusual interpreter bind/accept/egress
  SIDs) mark their associated 5-tuple / PID as a **flow of interest** and fire
  a capture trigger.
- Triggers are debounced per session so one incident does not spawn redundant
  captures.

### 4.2 Pre-trigger ring buffer (approved)

The interesting packets precede the alert. Sentinel maintains a **small,
in-memory rolling capture** *only for flows already flagged as of-interest*
(interpreter-owned sockets, new listeners, egress to public IPs) — never every
packet on the host. On trigger, the ring buffer is flushed to evidence so the
capture includes the **preceding context**, then capture continues for a
bounded window. Buffer is byte- and time-capped per flow; oldest data is
overwritten.

### 4.3 Tight scoping and hard caps (approved, mandatory)

- **Scope:** capture is restricted to the offending 5-tuple / peer IP / owning
  PID. No host-wide capture.
- **Caps:** per-capture time window, per-capture byte ceiling, global
  max-concurrent-captures, and a global disk-retention ceiling. A hostile flow
  must not be able to turn Sentinel's own sniffer into a disk-fill or
  memory-exhaustion DoS.
- **Fail-safe:** if `CAP_NET_RAW`/libpcap/AF_PACKET is unavailable, log the
  reason and degrade to **metadata-only** evidence (5-tuples and process
  linkage from `/proc` + eBPF). Capture failure must never disable detection —
  mirroring the existing bcc/eBPF fail-safe posture.

### 4.4 DPI enrichment (assimilated from Suricata)

Within the captured, scoped flow only, Sentinel reassembles and runs app-layer
heuristics to enrich the incident: **TLS SNI + JA4**, **DNS** queries/answers,
**HTTP** host/URI, **QUIC**, and generic protocol/anomaly detection. Output is
**EVE-style event records** (Section 3, `event_type` = `tls`/`dns`/`http`/…)
attached to the incident.

### 4.5 Evidence and audit

- Writes a scoped `.pcap` plus parsed EVE-style events under the incident's
  evidence directory; attaches them to the snapshot and incident.
- Appends a **capture audit entry** (who/what/when/scope/caps) analogous to
  `response-audit.log`.
- Capture events carry their own `sid` range and `classtype`
  (`evidence.capture`).

### 4.6 Privacy / threat-model gating (approved)

Payload capture is sensitive (can contain credentials, PII). It remains
evidence-preservation (read-only observation, consistent with the
explicit-not-silent stance) but is gated:

- **Opt-in config** (`capture.enabled`), default off.
- **Payload tier gating:** headers/metadata-only vs. full-payload as separate
  config levels.
- **Retention TTL** on captured artifacts; audited deletion.
- **Documented** in [THREAT_MODEL.md](THREAT_MODEL.md): what is captured, the
  privacy exposure, and the operator's legal/consent responsibility.
- **No silent enable**, no capture outside configured interface/scope.

## 5. What NOT to assimilate (YAGNI / non-goals)

- **No line-rate inline data plane.** No always-on full-traffic DPI, no packet
  verdicting/dropping at multi-gigabit.
- **No high-speed capture backends.** No DPDK, AF_XDP, netmap, PF_RING,
  Napatech, or RSS/fanout scaling. Triggered libpcap/AF_PACKET is sufficient.
- **No transparent bridge/tap deployment.** Sentinel watches the host's own
  traffic, not a span port between other devices.
- **No C ABI / embeddable-library surface** (`libsuricata` analogue).
- **No distributed sensor mesh here.** Fleet/collector work stays in its own
  design (`FLEET_DESIGN.md`), not this document.
- **No stealth, self-hiding, or automatic destructive remediation** — unchanged
  from Sentinel's standing rules.

## 6. Phased roadmap

Each phase preserves the oracle discipline: bash + Python remain authoritative
until Go reaches per-subsystem parity against the red-team harness, and every
signature keeps `sid` / `signature` / `classtype` / tests / docs.

- **Phase 0 — Schema-first.** Freeze and extend the v1 EVE-style event envelope
  in [SCHEMAS.md](SCHEMAS.md) (`event_type` discriminator; alert/incident/status
  as event types). This is the Python↔Go contract. *No language change yet.*
- **Phase 1 — Go agent skeleton.** Config, daemon sweep loop, `SystemState`
  equivalent, poll detectors ported, JSON output matching schemas. Runs
  alongside Python; harness compares both. *In progress:* the isolated
  `go-agent/` sidecar now has stdlib config loading, an injectable `/proc`
  process view, the JSONL sweep loop, and fixture parity for `deleted_exe`;
  Python remains the production agent.
- **Phase 2 — eBPF via `cilium/ebpf`.** Port exec/syscall rules; drop the
  bcc/Python runtime dependency; single static binary.
- **Phase 3 — Rule-engine parity + assimilation.** `rev`/`reference`/`metadata`,
  statebits (flowbits), thresholding, suppression, and — if rule count warrants
  — the MPM prefilter. `rules list/show/test/docs` parity.
- **Phase 4 — Session/correlation model.** Unify exec + network + FIM +
  rootcheck into sessions; re-project incidents over sessions.
- **Phase 5 — Network awareness + reactive forensic capture & DPI.** Ring
  buffer, triggered scoped capture, app-layer enrichment (JA4/DNS/TLS/HTTP/QUIC),
  datasets/reputation, evidence + audit + privacy gating.
- **Phase 6 — Framework/plugin surface.** Go interfaces for detectors, rules,
  response actions, and notify backends; administration-hub features; console
  parity.

## 7. Constraints preserved

- Detection stays useful without a dashboard, collector, or network.
- Read-only detection and dry-run/reviewed response until schemas and audit
  logs are tested and documented; **no automatic inline blocking** without an
  explicit, reviewed, audited `--apply` path.
- Every new detector, rule, response action, or capture event needs a stable
  `sid`, `signature`, `classtype`, tests, and operator docs.
- No stealth, self-hiding, persistence tricks, or automatic destructive
  remediation.
- The bash prototype and red-team harness remain the behavioral oracle through
  the migration.

## 8. Acceptance criteria

1. A written, reviewed schema (Phase 0) that both Python and a Go prototype can
   emit and that passes existing compatibility tests.
2. For each ported subsystem, Go output matches the Python/bash oracle on every
   relevant `sid` in the red-team harness before cutover.
3. Triggered capture demonstrably: (a) includes pre-trigger context from the
   ring buffer, (b) never exceeds configured time/byte/concurrency/retention
   caps under a flood, (c) degrades to metadata-only when capture privileges are
   absent, (d) is disabled by default and audited when enabled.
4. Every assimilated rule feature (statebits, thresholding, suppression) has
   fixture/red-team coverage and generated `RULES.md` documentation.

## 9. Open questions

- Go eBPF: `cilium/ebpf` (CO-RE, recommended) vs. a thin cgo shim to existing
  probes during transition?
- Capture library: `gopacket` + AF_PACKET vs. an eBPF/`AF_PACKET` custom path
  for tighter cap enforcement?
- Does the session model supersede or wrap the current `incident.py` grouping,
  and what is the migration for persisted incidents?
- Dataset/reputation feed sourcing and update policy (offline-first, signed).

## 10. Documentation impact (when work begins)

Per Sentinel's doc checklist, phased work must update: `SPECIFICATION.md`
(product/data model), `ROADMAP.md` (sequencing), `SCHEMAS.md` (event envelope),
`THREAT_MODEL.md` (capture privacy boundary), `COMMAND_REFERENCE.md` (capture +
rule commands), `RULES.md` (new SIDs), `RUNBOOKS.md` (capture-evidence
workflow), `config/enodia-sentinel.toml` (`capture.*` keys), and `INDEX.md` +
`VERSION.json` (versioning).