d835386381
Adds sidecar-only, --state-dir-gated FIM, package-DB-anchor, and rootcheck
engines to the Go migration sidecar, wired into agent.Sweep/Initialize
behind the existing baseline lifecycle. Adds a read-only, schema-checked
Python dashboard consumer (go_sidecar_state_dir, /api/go-sidecar/*, Sidecar
tab) that never starts, stops, or mutates the Go sidecar's state.
Also fixes drift found while reconciling this work: enodia_sentinel/web.py
had reinvented local schema constants instead of using the canonical
enodia_sentinel/schemas.py catalog (now registers enodia.go.sidecar.v1
there and reuses ALERT_SNAPSHOT_V1/INCIDENT_V1); docs/RULES.md had drifted
from what `rules docs` actually generates for SIDs 100069-100078 (ruleops.py
was missing metadata for four SIDs and had stale drill text for four more),
now back in sync with a regression test pinning them together.
Updates CLAUDE.md, README.md, go-agent/README.md, and docs/{ROADMAP,
GO_PORT_HANDOFF,SURICATA_ASSIMILATION,THREAT_MODEL,OPERATIONS,SCHEMAS,
COMMAND_REFERENCE}.md to reflect the landed work.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NQivSKBqQJsayz1xcYqWzZ
215 lines
10 KiB
Markdown
215 lines
10 KiB
Markdown
# Enodia Sentinel Go Agent (Migration Sidecar)
|
||
|
||
This directory is the parallel Go implementation described in
|
||
`docs/SURICATA_ASSIMILATION.md`. Python remains the production agent and the
|
||
behavioral oracle until every subsystem reaches fixture and red-team parity.
|
||
|
||
Current scope:
|
||
|
||
- standard-library-only flat TOML loading for the settings used so far;
|
||
- an injectable `/proc` process/file-descriptor/memory-map snapshot plus
|
||
fail-open `ss` socket collection;
|
||
- a cancellable sweep loop that emits JSONL `enodia.event.v1` records;
|
||
- exact `enodia.alert.v1` parity for `ld_preload` (SID `100011`),
|
||
`deleted_exe` (`100012`), `input_snooper` (`100032`), and
|
||
`credential_access` (`100033`), plus `reverse_shell` (`100010`), `egress`
|
||
(`100016`), `stealth_network` (`100036`), and the `/proc/<pid>/maps`
|
||
`memory_obfuscation` cohort (`100039`, `100048`, `100049`), and
|
||
`first_seen` (`100074`, `100075`), `new_listener` (`100013`),
|
||
`persistence` (`100015`), and `new_suid` (`100014`);
|
||
- an opt-in live baseline lifecycle (`--state-dir`) that builds listener/SUID
|
||
baselines, honors the startup grace gate, scans persistence paths, and
|
||
atomically retains `first-seen.json` using the Python-compatible schema;
|
||
- a sidecar-only FIM engine, enabled only with `--state-dir`, that persists a
|
||
SHA-256 security-path baseline as `fim-baseline.json`, performs
|
||
non-overlapping background comparisons, and can opt in to a bounded
|
||
`pacman -Qkk` package verification pass with `fim_pkg_verify = true`;
|
||
- a sidecar-only rootcheck engine that bounds PID cross-view probing with
|
||
`rootcheck_pid_cap`, runs one background comparison per interval, and checks
|
||
procfs, sysfs, `ps`, socket, module-taint, kernel-taint, and promiscuous-mode
|
||
views without blocking the polling loop;
|
||
- a sidecar-only package-DB anchor (`pkgdb_verify`) that fingerprints pacman's
|
||
local checksum database and alerts only on changes that lack a corresponding
|
||
transaction-log record; it never refreshes an unexplained changed anchor;
|
||
- pure exec and syscall event models plus replayable rule engines: built-in
|
||
exec SIDs `100001`–`100004`, configured `[[exec_rules]]`, and syscall SIDs
|
||
`100060`–`100066` now emit Python-identical alert envelopes without requiring
|
||
a kernel probe;
|
||
- typed host-event SIDs `100067`–`100073` and `100076`–`100078`, a unified
|
||
mixed JSONL pipeline, Python-compatible `rules-list`/`rules-show` metadata,
|
||
correlation SID `100080`, and a shared per-key cooldown gate;
|
||
- opt-in native exec (`--ebpf-exec`) and security-syscall (`--ebpf-syscall`)
|
||
tracepoint sources built with `cilium/ebpf` v0.22.0 and CO-RE parent-process
|
||
lookup. They feed the same rule/cooldown/envelope paths as replay and degrade
|
||
to polling when loading, attaching, or reading is unavailable. The syscall
|
||
probe filters its security-relevant syscall numbers in-kernel and also routes
|
||
`setuid`/`setgid` into typed host-event SIDs `100071`/`100072`, plus
|
||
`chmod`/`chown` variants into SID `100070`, `capset` into SID `100077`, and
|
||
`finit_module` into SID `100078` after resolving the module FD path. Confirmed
|
||
interpreter `write`, `writev`, `pwrite64`, `pwritev`, and `pwritev2` calls are
|
||
enter/exit-correlated for SID `100069`.
|
||
Successful IPv4/IPv6 `connect`, `bind`, `listen`, and `accept` calls feed SIDs
|
||
`100067`, `100073`, `100068`, and `100076`;
|
||
- `enodia.status.v1` heartbeat records whose alert totals, severity counts, and
|
||
last-alert time are derived from the retained Go snapshot store;
|
||
- optional bounded JSONL retention (`--event-log`) with private permissions,
|
||
one rotated segment, and synchronous flushes for non-status records;
|
||
- optional `enodia.alert.snapshot.v1` JSON/text pairs (`--snapshot-dir`) with
|
||
same-second alert batching, best-effort `/proc` context, and Python-compatible
|
||
count/age pruning;
|
||
- Python-compatible `enodia.incident.v1` grouping and atomic `incidents.json`
|
||
persistence, using process lineage first and the shared time-window fallback,
|
||
including additive SID `100080` correlation evidence;
|
||
- a shared fixture checked against the Python detector by
|
||
`scripts/check-go-parity.py`.
|
||
|
||
It does not yet provide Python's full enrichment or notification fan-out, and
|
||
it does not replace `enodia-sentinel`. Python's optional `/api/go-sidecar/*`
|
||
consumer can display this sidecar's isolated retained evidence, but remains
|
||
read-only and never starts or controls the Go process. The module requires Go
|
||
1.25 or newer; development is currently verified with Go 1.26.5. Run a single
|
||
terminal-visible stateless sweep:
|
||
|
||
```bash
|
||
cd go-agent
|
||
GOCACHE=/tmp/enodia-go-cache go run ./cmd/enodia-sentinel-go --once
|
||
```
|
||
|
||
Print the packaged agent identity with `enodia-sentinel-go --version`.
|
||
|
||
Exercise the live baseline lifecycle in an isolated directory (do not point the
|
||
migration sidecar at the production Python agent's state directory):
|
||
|
||
```bash
|
||
GOCACHE=/tmp/enodia-go-cache go run ./cmd/enodia-sentinel-go \
|
||
--state-dir /tmp/enodia-sentinel-go-state
|
||
```
|
||
|
||
Development verification from the repository root:
|
||
|
||
```bash
|
||
make generate-go # only after editing internal/ebpfsource/bpf/*.bpf.c
|
||
make test-go
|
||
make parity-go
|
||
```
|
||
|
||
Build and install the opt-in production validation sidecar from the repository
|
||
root:
|
||
|
||
```bash
|
||
make build-go
|
||
sudo make install-go
|
||
sudo make enable-go
|
||
make status-go
|
||
make logs-go
|
||
```
|
||
|
||
`enodia-sentinel-go-sidecar.service` writes JSONL to the system journal and
|
||
keeps baseline lifecycle files under `/var/lib/enodia-sentinel-go`. It also
|
||
retains the same envelopes in `events.jsonl`, rotating to `events.jsonl.1`
|
||
before the active file exceeds 64 MiB. Both files are mode `0600`; alert and
|
||
incident records are synchronized before emission succeeds, while routine
|
||
status records use normal kernel writeback. It never shares the Python daemon's
|
||
`/var/log/enodia-sentinel` state. Both native probes are requested by the unit;
|
||
a probe load failure is visible in status events and the polling loop remains
|
||
active. Installing the sidecar does not enable it or alter
|
||
`enodia-sentinel.service`. The unit reaches `active` only after baseline
|
||
initialization, event-log and snapshot-store preflight, and both requested probe
|
||
load attempts finish. Snapshot-store preflight includes the private assurance
|
||
chain.
|
||
|
||
Alert envelopes are also batched into private
|
||
`alert-YYYYMMDD-HHMMSS.{json,log}` pairs under the same isolated directory.
|
||
The JSON side implements the required `enodia.alert.snapshot.v1` fields and the
|
||
service applies shared `max_snapshots` / `max_snapshot_age_days` limits. Status
|
||
counts are populated only from parseable retained snapshots. Each snapshot is
|
||
linked to an incident in a private atomic `incidents.json` index. Grouping uses
|
||
the configured `incident_tracking`, `incident_window`, and
|
||
`incident_lineage_depth` settings; same-second alert batches update one incident
|
||
and one snapshot name idempotently. Hot-path enrichment derives bounded
|
||
process/parent context, lineage, remote-IP classification, and candidate paths
|
||
from already-captured data without new I/O. A 16-job asynchronous worker then
|
||
adds streamed SHA-256 values for regular executables up to 8 MiB, file metadata,
|
||
and pacman/dpkg/rpm package-owner strings with a two-second query bound;
|
||
saturation leaves fields unknown rather than delaying alerts. The sidecar also
|
||
has separate, bounded FIM/package-verification, package-DB-anchor, and
|
||
rootcheck engines; notification fan-out remains future parity work. The worker
|
||
reports truthful FIM-baseline, package-DB-anchor, and Go hash-chain
|
||
presence/age, plus the configured `pacman -Qkk` and rootcheck cadences. This is
|
||
metadata only: recording a snapshot never starts integrity work.
|
||
Snapshot text/JSON
|
||
revisions append synchronized `enodia.hash_chain.v1`
|
||
records to a private local `hash-chain.jsonl`.
|
||
|
||
After every successfully emitted status record, the service atomically updates
|
||
`/var/lib/enodia-sentinel-go/heartbeat` with the current Unix timestamp (mode
|
||
`0600`). The file remains after shutdown so external watchdogs can identify a
|
||
stale sensor.
|
||
|
||
Check that signal with JSON output and a watchdog-friendly exit status:
|
||
|
||
```bash
|
||
enodia-sentinel-go --health
|
||
# For an isolated development run:
|
||
enodia-sentinel-go --health --state-dir /tmp/enodia-sentinel-go-state
|
||
```
|
||
|
||
The command uses `heartbeat_max_age` from the shared configuration, defaults to
|
||
the service state path when `--state-dir` is omitted, and exits nonzero for a
|
||
missing, malformed, or stale heartbeat.
|
||
|
||
For an ad-hoc retained run, set an explicit path and byte bound:
|
||
|
||
```bash
|
||
enodia-sentinel-go --state-dir /tmp/enodia-go-state \
|
||
--event-log /tmp/enodia-go-state/events.jsonl \
|
||
--event-log-max-bytes 8388608
|
||
```
|
||
|
||
Inspect recent retained records across both the active and rotated segments:
|
||
|
||
```bash
|
||
sudo enodia-sentinel-go --events-tail 50
|
||
# Or inspect an ad-hoc state tree:
|
||
enodia-sentinel-go --events-tail 50 --state-dir /tmp/enodia-go-state
|
||
```
|
||
|
||
The reader returns chronological JSONL, validates every source record, and
|
||
fails nonzero rather than forwarding corrupted evidence. It does not load the
|
||
agent configuration, so retained evidence remains available while a broken
|
||
configuration prevents the service from starting.
|
||
|
||
Inspect the isolated incident index without starting the agent:
|
||
|
||
```bash
|
||
enodia-sentinel-go --incidents-list --state-dir /tmp/enodia-go-state
|
||
enodia-sentinel-go --incident-show inc-20260722-101112-abcd --state-dir /tmp/enodia-go-state
|
||
```
|
||
|
||
`--incident-show` returns `enodia.incident.view.v1` with the durable incident,
|
||
available snapshot reports, and a time-ordered timeline. These read-only
|
||
commands never modify or consult the Python daemon state.
|
||
|
||
`--fixture`, `--host`, `--timestamp`, `--proc-root`, and `--suid-root` exist for
|
||
deterministic parity/testing. Fixture mode never activates the live baseline
|
||
manager. Production work should continue to use the default live `/proc`,
|
||
hostname, and local timestamp.
|
||
|
||
The kernel sources are explicitly opt-in while this remains a sidecar. They
|
||
require the privileges needed to load BPF programs and create maps. A load
|
||
failure is reported on stderr and in `enodia.status.v1`, then the poll loop
|
||
continues:
|
||
|
||
```bash
|
||
sudo go run ./cmd/enodia-sentinel-go --ebpf-exec --ebpf-syscall
|
||
```
|
||
|
||
Offline event-rule replays remain available without BPF or root:
|
||
|
||
```bash
|
||
go run ./cmd/enodia-sentinel-go --exec-events ../tests/fixtures/go/exec-events.json
|
||
go run ./cmd/enodia-sentinel-go --syscall-events ../tests/fixtures/go/syscall-events.json
|
||
go run ./cmd/enodia-sentinel-go --host-events ../tests/fixtures/go/host-events.json
|
||
go run ./cmd/enodia-sentinel-go --event-stream ../tests/fixtures/go/mixed-events.jsonl
|
||
go run ./cmd/enodia-sentinel-go --rules-list
|
||
```
|