enodia-sentinal/docs/SURICATA_ASSIMILATION.md

Suricata Assimilation Design

Status: Draft / design proposal Date: 2026-07-10 Applies to package line: 0.7.x (forward-looking; no shipped behavior yet) Owner doc: sits alongside SPECIFICATION.md and ROADMAP.md

1. Purpose and framing

This document specifies how Enodia Sentinel should assimilate the valuable ideas of Suricata — a mature network IDS/IPS/NSM engine — into an integrated host security and OS-state monitoring, administration hub, and framework that unifies EDR and (host-scoped) IDS/IPS.

The guiding principle is idea donor, not engine port. We take Suricata's proven models (rule language, flow/session correlation, app-layer transaction parsing, unified event schema, multi-pattern prefiltering, anomaly events, datasets/reputation) and re-express them in Sentinel's own model — every signature keeping a stable sid, signature, and classtype, with tests and operator docs — without rebuilding Suricata's line-rate packet data plane.

It also records the recommended language migration to Go and a new first-class capability: reactive, triggered forensic packet capture and DPI that activates when a cyberattack is detected against the host.

1.1 Target identity (load-bearing assumption)

Enodia Sentinel is a host-based platform: HIDS/HIPS + EDR + OS-state administration, with network awareness of the host's own traffic. It is not a transparent inline network appliance placed between routers doing always-on multi-gigabit reassembly and verdicting. Every decision below flows from that assumption; if the identity changes to a line-rate network sensor, Section 2 (language) and Section 5 (non-goals) must be revisited.

A true line-rate network appliance (tap/bridge role, inspecting traffic that is not the host's own) is explicitly deferred to a possible future sibling project, not a mode of Sentinel. It is not a current priority. This keeps Sentinel's host-scoped identity — and therefore the Go decision — clean, and is where the Rust escape hatch of Section 2.1 would live if that sibling is ever built.

2. Language and migration decision

2.1 Recommendation: migrate to Go, keep Rust as an escape hatch

For an integrated host hub and framework, the dominant workload is orchestration: polling OS state, loading eBPF, correlating events, serving a console, and performing reviewed administration actions. That is Go's sweet spot. Decisive factors:

• eBPF without a runtime toolchain. Today the event layer shells to bcc, which drags Python + clang into the runtime. Go's cilium/ebpf is a pure-Go CO-RE loader: one static binary, no bcc, no clang. This removes Sentinel's largest deployment dependency and matches the "useful without extras, easy to deploy" ethos.
• Concurrency model. Goroutines map cleanly onto per-flow, per-detector, and per-connection work.
• Framework surface. A plugin/extension framework wants stable interfaces and a single auditable binary — both natural in Go.
• Velocity. A broad-scope platform needs fast iteration and a low contributor barrier.

Rust would only win if Sentinel committed to line-rate inline network packet inspection — which the identity statement explicitly rejects. Suricata adopted Rust for its untrusted-packet data plane; Sentinel has no such data plane and therefore does not inherit that requirement. If a hot untrusted-input parsing core ever appears (e.g. a bespoke protocol decoder proves to be a bottleneck), introduce a small Rust component behind a stable interface then — do not pay the whole-rewrite tax now.

2.2 The dependency-discipline caveat

Sentinel's Python core is stdlib-only. A Go rewrite cannot be literally zero-dependency: practical eBPF and packet capture require vetted libraries (cilium/ebpf, gopacket). The ethos is preserved as "few, audited, security-relevant dependencies, one static binary" rather than pure stdlib. This is an explicit, documented tradeoff, not a drift. The console (net/http + crypto/tls), JSON (encoding/json), and config remain stdlib.

2.3 Migration strategy: strangler via stable schemas

The Python implementation is not thrown away; it becomes the behavioral oracle during transition, exactly as the bash prototype (src/sentinel.sh) is the oracle for the Python port today.

• Schemas are the contract. The v1 JSON schemas in SCHEMAS.md (enodia.alert.v1, enodia.incident.v1, enodia.status.v1, …) define the interface between Python and Go. The Go agent must emit byte-comparable structured output for the same inputs.
• One red-team harness, three implementations. The existing red-team drill harness runs against bash, Python, and (incrementally) Go. A subsystem is "cut over" only when Go matches the oracle on every relevant sid.
• Subsystem-by-subsystem. Ported in the phase order of Section 6; Python keeps ownership of anything not yet at parity. No big-bang switch.

3. Suricata feature-assimilation catalog

Each row is a Suricata idea, its value, and how it maps into Sentinel's model. "Keep" = adopt; "adapt" = adopt with host-scoping; "skip" = out of scope (Section 5).

Suricata concept	Value	Sentinel mapping	Verdict
Rule metadata: sid, rev, msg, classtype, priority, reference, metadata	Stable, referenceable, tunable signatures	Sentinel already has sid/signature/classtype/severity; add rev, reference, metadata to the rule model in ruleops.py	Keep
flowbits (set/isset/unset/toggle/noalert)	Stateful correlation across events in one flow	A "statebits" facility on the session object (Section 3.1) so a rule can fire only after a prior condition in the same session	Adapt
threshold / detection_filter (limit, threshold, both; by src/dst)	Rate-limits noisy signatures	Per-sid thresholding in the rule engine; mirror Suricata semantics (limit/threshold/both, per-scope, per-window)	Keep
suppress	Silences known-benign matches by scope	Extends the existing reconcile.py acknowledged-drift model to rule-level suppression by sid + scope	Adapt
Fast-pattern / MPM (Aho-Corasick) prefilter	Scales to thousands of content rules cheaply	An Aho-Corasick prefilter stage in front of full rule evaluation, over argv/paths/captured payloads — introduced only when rule count justifies it (YAGNI gate)	Adapt
Flow engine (bidirectional flow hash, state, timeouts)	The correlation backbone	A host-scoped session/flow model (Section 3.1) keyed on 5-tuple and owning PID lineage	Adapt
App-layer transactions + protocol detection	Structured per-request/response records	Applied only to triggered captures (Section 4), producing per-transaction records for enrichment	Adapt
EVE unified JSON (event_type: alert, flow, dns, tls, http, quic, ssh, fileinfo, anomaly)	One machine-readable envelope	Formalize schemas.py into an EVE-style event envelope with a typed event_type discriminator; existing alert/incident schemas become event types	Keep
JA3/JA4 TLS fingerprinting	Identify clients/malware from TLS ClientHello	JA4 computation over triggered TLS captures, attached to the incident	Keep
Decoder / anomaly events	"Malformed input is itself a signal"	Anomaly events from capture DPI (truncated/invalid app-layer) as first-class low-severity signals	Adapt
Datasets / datajson / iprep (bulk IOC + reputation)	Match against large IP/domain/hash sets	A dataset store feeding egress, DNS, and TLS signals; explicit IOC lists distinct from behavioral heuristics	Adapt
Filestore + file hashing	Extract and hash transferred files	Bounded extraction from triggered captures, hashed and attached as evidence; reuse provenance.py hashing conventions	Adapt
Stream (TCP) reassembly	Ordered bytes for app-layer parsing	Only within a captured flow for DPI — not an always-on reassembly engine	Adapt

3.1 The session/flow correlation model

Sentinel today is process/host-centric; Suricata is flow-centric. The assimilation is a session object that unifies both and realizes the roadmap's "correlation across exec, network, persistence, FIM, and rootcheck":

• Key: owning process lineage and any associated network 5-tuples.
• Aggregates: exec events, syscall events, network connections, FIM/pkg changes, rootcheck findings, and posture context observed for that lineage within a time window.
• Carries statebits: the flowbits-equivalent, enabling rules like "credential file opened then egress to a public IP in the same session."
• Feeds incidents: incident.py's current lineage+window grouping becomes a projection over sessions; snapshots attach session evidence.

4. Reactive forensic capture & DPI

A packet inspector and sniffer that activates when an attack is detected, preserving wire evidence the way Sentinel already preserves process/disk evidence. It is bounded and event-driven — not always-on inspection — which is why it does not change the Go language call.

4.1 Trigger model

• High-severity detections (reverse_shell, egress/C2, new_listener, stealth_network, C2 exec rules, unusual interpreter bind/accept/egress SIDs) mark their associated 5-tuple / PID as a flow of interest and fire a capture trigger.
• Triggers are debounced per session so one incident does not spawn redundant captures.

4.2 Pre-trigger ring buffer (approved)

The interesting packets precede the alert. Sentinel maintains a small, in-memory rolling capture only for flows already flagged as of-interest (interpreter-owned sockets, new listeners, egress to public IPs) — never every packet on the host. On trigger, the ring buffer is flushed to evidence so the capture includes the preceding context, then capture continues for a bounded window. Buffer is byte- and time-capped per flow; oldest data is overwritten.

4.3 Tight scoping and hard caps (approved, mandatory)

• Scope: capture is restricted to the offending 5-tuple / peer IP / owning PID. No host-wide capture.
• Caps: per-capture time window, per-capture byte ceiling, global max-concurrent-captures, and a global disk-retention ceiling. A hostile flow must not be able to turn Sentinel's own sniffer into a disk-fill or memory-exhaustion DoS.
• Fail-safe: if CAP_NET_RAW/libpcap/AF_PACKET is unavailable, log the reason and degrade to metadata-only evidence (5-tuples and process linkage from /proc + eBPF). Capture failure must never disable detection — mirroring the existing bcc/eBPF fail-safe posture.

4.4 DPI enrichment (assimilated from Suricata)

Within the captured, scoped flow only, Sentinel reassembles and runs app-layer heuristics to enrich the incident: TLS SNI + JA4, DNS queries/answers, HTTP host/URI, QUIC, and generic protocol/anomaly detection. Output is EVE-style event records (Section 3, event_type = tls/dns/http/…) attached to the incident.

4.5 Evidence and audit

• Writes a scoped .pcap plus parsed EVE-style events under the incident's evidence directory; attaches them to the snapshot and incident.
• Appends a capture audit entry (who/what/when/scope/caps) analogous to response-audit.log.
• Capture events carry their own sid range and classtype (evidence.capture).

4.6 Privacy / threat-model gating (approved)

Payload capture is sensitive (can contain credentials, PII). It remains evidence-preservation (read-only observation, consistent with the explicit-not-silent stance) but is gated:

• Opt-in config (capture.enabled), default off.
• Payload tier gating: headers/metadata-only vs. full-payload as separate config levels.
• Retention TTL on captured artifacts; audited deletion.
• Documented in THREAT_MODEL.md: what is captured, the privacy exposure, and the operator's legal/consent responsibility.
• No silent enable, no capture outside configured interface/scope.

5. What NOT to assimilate (YAGNI / non-goals)

• No line-rate inline data plane. No always-on full-traffic DPI, no packet verdicting/dropping at multi-gigabit.
• No high-speed capture backends. No DPDK, AF_XDP, netmap, PF_RING, Napatech, or RSS/fanout scaling. Triggered libpcap/AF_PACKET is sufficient.
• No transparent bridge/tap deployment. Sentinel watches the host's own traffic, not a span port between other devices.
• No C ABI / embeddable-library surface (libsuricata analogue).
• No distributed sensor mesh here. Fleet/collector work stays in its own design (FLEET_DESIGN.md), not this document.
• No stealth, self-hiding, or automatic destructive remediation — unchanged from Sentinel's standing rules.

6. Phased roadmap

Each phase preserves the oracle discipline: bash + Python remain authoritative until Go reaches per-subsystem parity against the red-team harness, and every signature keeps sid / signature / classtype / tests / docs.

• Phase 0 — Schema-first. Freeze and extend the v1 EVE-style event envelope in SCHEMAS.md (event_type discriminator; alert/incident/status as event types). This is the Python↔Go contract. No language change yet.
• Phase 1 — Go agent skeleton. Config, daemon sweep loop, SystemState equivalent, poll detectors ported, JSON output matching schemas. Runs alongside Python; harness compares both. In progress: the isolated go-agent/ sidecar now has stdlib config loading, an injectable /proc process/file-descriptor/memory-map and ss socket view, the JSONL sweep loop, and fixture parity for reverse_shell, ld_preload, deleted_exe, input_snooper, credential_access, stealth_network, the memory_obfuscation cohort, and egress; Python remains the production agent.
• Phase 2 — eBPF via cilium/ebpf. The transport-independent exec/syscall event models and rule engines have offline Python-envelope parity in Go. An opt-in native execve and filtered raw-syscall tracepoints now feed both rule engines through cilium/ebpf with CO-RE parent lookup and fail-open polling fallback; root UID/GID transitions and persistence-path permission changes plus capability-set and file-backed module-load requests also feed the typed-host engine. Interpreter write/writev/pwrite variants are enter/exit-correlated and filtered in-kernel before their FD paths are resolved in userspace. Successful connect/bind calls are likewise correlated, with TCP inode verification for connect events; listen/accept calls resolve both endpoints from the successful socket FD. An opt-in hardened systemd validation unit now runs the static Go binary with isolated baseline state, journal JSONL, a bounded retained JSONL stream, and an atomic health marker. All current typed-host families now have native transports. Bounded enodia.alert.snapshot.v1 JSON/text pairs now retain alerts with basic process context and feed truthful status counts. Incident grouping, rich enrichment/assurance, management-consumer integration, and broader parity must land before the bcc/Python runtime dependency can be dropped.
• Phase 3 — Rule-engine parity + assimilation. rev/reference/metadata, statebits (flowbits), thresholding, suppression, and — if rule count warrants — the MPM prefilter. rules list/show/test/docs parity.
• Phase 4 — Session/correlation model. Unify exec + network + FIM + rootcheck into sessions; re-project incidents over sessions.
• Phase 5 — Network awareness + reactive forensic capture & DPI. Ring buffer, triggered scoped capture, app-layer enrichment (JA4/DNS/TLS/HTTP/QUIC), datasets/reputation, evidence + audit + privacy gating.
• Phase 6 — Framework/plugin surface. Go interfaces for detectors, rules, response actions, and notify backends; administration-hub features; console parity.

7. Constraints preserved

• Detection stays useful without a dashboard, collector, or network.
• Read-only detection and dry-run/reviewed response until schemas and audit logs are tested and documented; no automatic inline blocking without an explicit, reviewed, audited --apply path.
• Every new detector, rule, response action, or capture event needs a stable sid, signature, classtype, tests, and operator docs.
• No stealth, self-hiding, persistence tricks, or automatic destructive remediation.
• The bash prototype and red-team harness remain the behavioral oracle through the migration.

8. Acceptance criteria

1. A written, reviewed schema (Phase 0) that both Python and a Go prototype can emit and that passes existing compatibility tests.
2. For each ported subsystem, Go output matches the Python/bash oracle on every relevant sid in the red-team harness before cutover.
3. Triggered capture demonstrably: (a) includes pre-trigger context from the ring buffer, (b) never exceeds configured time/byte/concurrency/retention caps under a flood, (c) degrades to metadata-only when capture privileges are absent, (d) is disabled by default and audited when enabled.
4. Every assimilated rule feature (statebits, thresholding, suppression) has fixture/red-team coverage and generated RULES.md documentation.

9. Open questions

• Go eBPF: cilium/ebpf (CO-RE, recommended) vs. a thin cgo shim to existing probes during transition?
• Capture library: gopacket + AF_PACKET vs. an eBPF/AF_PACKET custom path for tighter cap enforcement?
• Does the session model supersede or wrap the current incident.py grouping, and what is the migration for persisted incidents?
• Dataset/reputation feed sourcing and update policy (offline-first, signed).

10. Documentation impact (when work begins)

Per Sentinel's doc checklist, phased work must update: SPECIFICATION.md (product/data model), ROADMAP.md (sequencing), SCHEMAS.md (event envelope), THREAT_MODEL.md (capture privacy boundary), COMMAND_REFERENCE.md (capture + rule commands), RULES.md (new SIDs), RUNBOOKS.md (capture-evidence workflow), config/enodia-sentinel.toml (capture.* keys), and INDEX.md + VERSION.json (versioning).