feat(go): advance validation sidecar toward production
This commit is contained in:
parent
6a06eba255
commit
f85c2e831a
92 changed files with 8881 additions and 91 deletions
145
docs/GO_PORT_HANDOFF.md
Normal file
145
docs/GO_PORT_HANDOFF.md
Normal file
|
|
@ -0,0 +1,145 @@
|
|||
# Go Port Handoff
|
||||
|
||||
Saved: 2026-07-22T01:40:52-07:00
|
||||
Branch: `main`
|
||||
Base commit: `6a06eba` (`docs(behavior): add behavioral spec for sweep + baseline lifecycle`)
|
||||
Status: implemented and green, but uncommitted
|
||||
|
||||
## Worktree warning
|
||||
|
||||
The checkout is intentionally dirty and contains work from multiple related
|
||||
continuations. Do not reset, clean, or broadly restage it.
|
||||
|
||||
- At this checkpoint, `git status --short` has 62 entries (105 when every
|
||||
untracked file is expanded instead of collapsing directories).
|
||||
- This handoff is itself currently untracked and should be included when the
|
||||
Go migration tranche is eventually committed.
|
||||
- `docs/behavior/02-poll-detectors.md` and `docs/behavior/index.md` already had
|
||||
staged content plus later unstaged edits (`AM` / `MM`). This save did not
|
||||
change the index.
|
||||
- The Python GUI files and tests are separate pre-existing work. Preserve them
|
||||
while continuing the Go port.
|
||||
- `build/` is ignored and contains only local build output.
|
||||
|
||||
## Implemented Go surface
|
||||
|
||||
The Go module requires Go 1.25 or newer and is currently verified with
|
||||
`go1.26.5-X:nodwarf5`. It uses `github.com/cilium/ebpf` v0.22.0 and builds as a
|
||||
static binary.
|
||||
|
||||
### Polling and lifecycle
|
||||
|
||||
- All current Python poll detectors have Go parity: `reverse_shell`,
|
||||
`ld_preload`, `deleted_exe`, `input_snooper`, `credential_access`,
|
||||
`stealth_network`, `memory_obfuscation`, `egress`, `first_seen`,
|
||||
`new_listener`, `persistence`, and `new_suid`.
|
||||
- `--state-dir` enables isolated listener/SUID baselines, startup grace,
|
||||
persistence scans, and atomic first-seen retention.
|
||||
- The Go state directory must never be the Python daemon's
|
||||
`/var/log/enodia-sentinel` directory.
|
||||
|
||||
### Event rules and native sources
|
||||
|
||||
- Exec rules: built-in SIDs `100001`-`100004` plus configured rules.
|
||||
- Syscall rules: SIDs `100060`-`100066`.
|
||||
- Typed host rules: SIDs `100067`-`100073` and `100076`-`100078`.
|
||||
- Mixed JSONL routing, rule catalog operations, correlation SID `100080`, and
|
||||
one shared cooldown gate are implemented.
|
||||
- Embedded CO-RE exec and filtered raw-syscall probes use `cilium/ebpf` and
|
||||
fail open to polling on load/attach/read failures.
|
||||
- Native typed transports include UID/GID changes, chmod/chown variants,
|
||||
`capset`, `finit_module`, confirmed interpreter `write`/`writev`/`pwrite64`/
|
||||
`pwritev`/`pwritev2`, and successful TCP connect/bind/listen/accept calls
|
||||
with `/proc` endpoint resolution.
|
||||
- Generated little- and big-endian BPF objects live under
|
||||
`go-agent/internal/ebpfsource/`. Run `make generate-go` only after changing
|
||||
`go-agent/internal/ebpfsource/bpf/*.bpf.c`.
|
||||
|
||||
### Production validation path
|
||||
|
||||
- `make build-go` creates a stripped, static, VCS-independent binary.
|
||||
- `make install-go` installs the separate binary and
|
||||
`enodia-sentinel-go-sidecar.service` without replacing or enabling the
|
||||
Python service.
|
||||
- The hardened systemd unit has a current `systemd-analyze security` exposure
|
||||
score of 4.3 (`OK`). It preserves the host `/proc`, `/tmp`, and network view
|
||||
because detectors require them, while hiding the Python state tree.
|
||||
- The unit uses `Type=notify`; READY is sent only after baseline setup, event
|
||||
log preflight, and both requested probe-load attempts.
|
||||
- JSONL remains visible in journald and is also retained as private bounded
|
||||
`/var/lib/enodia-sentinel-go/events.jsonl` plus `events.jsonl.1` (64 MiB per
|
||||
active segment boundary). Non-status events are synchronized to disk.
|
||||
- `/var/lib/enodia-sentinel-go/heartbeat` is atomically refreshed after each
|
||||
successful status emission.
|
||||
- `enodia-sentinel-go --health` returns `enodia.go.health.v1` JSON and a
|
||||
watchdog-friendly exit status.
|
||||
- `enodia-sentinel-go --events-tail N` reads active and rotated retained events
|
||||
chronologically, validates JSON, fails closed on corruption, and deliberately
|
||||
works without loading the agent configuration.
|
||||
- The unit now retains private `alert-YYYYMMDD-HHMMSS.{json,log}` pairs using
|
||||
the required `enodia.alert.snapshot.v1` fields. Same-second alerts merge into
|
||||
one pair, process context is collected best-effort from `/proc`, and shared
|
||||
`max_snapshots` / `max_snapshot_age_days` limits are applied at startup,
|
||||
capture, and status time.
|
||||
- `enodia.status.v1` totals, severity counts, and last-alert time are populated
|
||||
only from parseable retained Go snapshots. Python's unchanged `list_alerts`
|
||||
and `get_alert` data layer successfully consumed a Go-produced pair.
|
||||
- Snapshot incident IDs remain null and enrichment currently contains only a
|
||||
Go-source marker plus process detail; incident indexing, rich enrichment,
|
||||
assurance chaining, and notification fan-out are not yet ported.
|
||||
- No live system service was installed or enabled during development.
|
||||
|
||||
## Last green verification
|
||||
|
||||
Run from the repository root:
|
||||
|
||||
```bash
|
||||
python3 -m unittest discover -s tests -q
|
||||
cd go-agent
|
||||
GOCACHE=/tmp/enodia-go-cache go test ./...
|
||||
GOCACHE=/tmp/enodia-go-cache go test -race ./...
|
||||
GOCACHE=/tmp/enodia-go-cache go vet ./...
|
||||
go mod verify
|
||||
cd ..
|
||||
make parity-go
|
||||
make check-go-service
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Results at save time:
|
||||
|
||||
- Python: 377 tests passed.
|
||||
- Go unit and race suites: all packages passed.
|
||||
- `go vet` and `go mod verify`: passed.
|
||||
- Parity: 22 poll alerts, 5 exec alerts, 7 syscall alerts, 10 typed-host
|
||||
alerts, 22 catalog records, 3 mixed-stream alerts, and 2 correlations.
|
||||
- `systemd-analyze verify` and the three Go sidecar packaging/isolation tests:
|
||||
passed.
|
||||
- Static builds: `linux/amd64`, `linux/arm64`, `linux/riscv64`, and
|
||||
`linux/s390x` passed.
|
||||
- Integration smoke tests passed for static binary identity, JSON emission,
|
||||
READY/STOPPING notification, heartbeat freshness/staleness, retained event
|
||||
equivalence, tail selection, corruption rejection, and evidence reading with
|
||||
malformed TOML. Snapshot smoke tests also passed for same-second batching,
|
||||
severity aggregation, private JSON/text pairs, truthful status counts,
|
||||
startup/quiet-period retention, process context, and Python dashboard-reader
|
||||
compatibility.
|
||||
|
||||
The Python suite prints three expected CLI error lines from negative tests and
|
||||
one existing `ResourceWarning` for an implicitly cleaned-up HTTP 401 object;
|
||||
the suite still exits successfully.
|
||||
|
||||
## Resume here
|
||||
|
||||
1. Add Python-compatible incident grouping/index persistence so retained Go
|
||||
snapshots can carry real `incident_id` values and correlation timelines.
|
||||
2. Port bounded post-alert enrichment and local assurance chaining without
|
||||
adding destructive response behavior or blocking the event loop on slow
|
||||
collectors.
|
||||
3. Connect the isolated Go snapshot/event state to explicit read-only
|
||||
management consumers while keeping Python authoritative.
|
||||
4. Continue broader Phase 3 rule metadata/state parity before considering any
|
||||
default-service or package cutover.
|
||||
|
||||
Keep the validation sidecar opt-in, preserve its separate state tree, and keep
|
||||
new attack/event primitives disconnected from destructive response paths.
|
||||
|
|
@ -62,6 +62,65 @@ sudo systemctl daemon-reload
|
|||
sudo systemctl restart enodia-sentinel.service
|
||||
```
|
||||
|
||||
## Go Migration Sidecar (opt-in)
|
||||
|
||||
The Go agent has a separate validation service and is never installed by the
|
||||
default `make install` target or the Arch package. Build and install it
|
||||
explicitly from a checkout with Go 1.25 or newer:
|
||||
|
||||
```bash
|
||||
make build-go
|
||||
sudo make install-go PREFIX=/usr/local
|
||||
sudo make enable-go
|
||||
```
|
||||
|
||||
The static `enodia-sentinel-go` binary emits `enodia.event.v1` JSONL to the
|
||||
system journal. Its service owns `/var/lib/enodia-sentinel-go` for baseline
|
||||
state and a bounded retained event stream. `events.jsonl` rotates to one
|
||||
`events.jsonl.1` segment before crossing 64 MiB; both are private to root, and
|
||||
non-status records are synchronized before emission succeeds. The service
|
||||
requests the embedded native exec and syscall probes and fails open to polling
|
||||
when the kernel refuses a probe. It does not write the production Python
|
||||
daemon's `/var/log/enodia-sentinel` state or replace
|
||||
`enodia-sentinel.service`. Systemd readiness is emitted only after baseline
|
||||
initialization, event-log and snapshot-store preflight, and both requested probe
|
||||
load attempts complete.
|
||||
|
||||
Alert events are additionally retained as private
|
||||
`alert-YYYYMMDD-HHMMSS.{json,log}` pairs in `/var/lib/enodia-sentinel-go`.
|
||||
Same-second events are merged into one snapshot, required
|
||||
`enodia.alert.snapshot.v1` fields are preserved, and the shared
|
||||
`max_snapshots` / `max_snapshot_age_days` settings bound retention. The current
|
||||
Go enrichment block is intentionally minimal; Python remains authoritative for
|
||||
incident grouping, rich enrichment, assurance chaining, and notifications.
|
||||
|
||||
Each successful sweep atomically refreshes
|
||||
`/var/lib/enodia-sentinel-go/heartbeat` as a Unix timestamp with mode `0600`;
|
||||
the retained value becomes stale when the sidecar stops or wedges.
|
||||
`enodia-sentinel-go --health` reports that state as JSON and exits nonzero when
|
||||
it is unavailable or older than the configured `heartbeat_max_age`.
|
||||
|
||||
Inspect the retained event stream without depending on journald:
|
||||
|
||||
```bash
|
||||
sudo tail -n 50 /var/lib/enodia-sentinel-go/events.jsonl
|
||||
sudo enodia-sentinel-go --events-tail 50
|
||||
```
|
||||
|
||||
The native reader includes `events.jsonl.1`, preserves chronological order, and
|
||||
fails nonzero if either retained segment contains malformed JSON.
|
||||
|
||||
Inspect or remove the validation service with:
|
||||
|
||||
```bash
|
||||
sudo make status-go
|
||||
sudo make disable-go
|
||||
sudo make uninstall-go
|
||||
```
|
||||
|
||||
Uninstalling preserves the shared configuration and sidecar state for forensic
|
||||
review or a later reinstall.
|
||||
|
||||
## Debian and Ubuntu
|
||||
|
||||
There is no native `.deb` package yet. Use the source install path:
|
||||
|
|
|
|||
|
|
@ -259,12 +259,39 @@ Exit criteria:
|
|||
|
||||
- Suricata model assimilation design is complete, and Phase 0's additive
|
||||
`enodia.event.v1` Python reference contract is implemented.
|
||||
- Phase 1 is in progress under `go-agent/`: a stdlib-only parallel sweep loop,
|
||||
injectable process/file-descriptor/socket state, and seven poll detectors
|
||||
(`reverse_shell`, `ld_preload`, `deleted_exe`, `input_snooper`,
|
||||
`credential_access`, `stealth_network`, `egress`) are checked against the
|
||||
Python oracle with a shared fixture. The Go sidecar remains stdout-only and
|
||||
non-production until broader parity lands.
|
||||
- Phase 1 under `go-agent/` now has a stdlib-only parallel sweep loop,
|
||||
injectable process/file-descriptor/socket state, and all current Python poll
|
||||
detectors ported: `reverse_shell`, `ld_preload`, `deleted_exe`,
|
||||
`input_snooper`, `credential_access`, `stealth_network`, `egress`,
|
||||
`memory_obfuscation`, `first_seen`, `new_listener`, `persistence`, and
|
||||
`new_suid`. The shared fixture and `scripts/check-go-parity.py` check Go
|
||||
output against the Python oracle. A new `provenance` package provides
|
||||
pacman/dpkg/rpm ownership queries so the Go `new_listener` detector can
|
||||
mirror Python's package-owned-listener suppression. Opt-in `--state-dir`
|
||||
mode now builds listener/SUID baselines, applies the shared startup grace
|
||||
gate, scans persistence paths, and retains first-seen network state. The Go
|
||||
sidecar also has offline-replay parity for the exec rule family (built-ins
|
||||
`100001`–`100004` plus configured `[[exec_rules]]`) and syscall rules
|
||||
`100060`–`100066`, plus all typed host rules, the active rule catalog,
|
||||
mixed-event JSONL routing, correlation SID `100080`, and sweep cooldown. The
|
||||
first Phase 2 transports have landed as an opt-in `cilium/ebpf` execve
|
||||
tracepoint source plus a filtered raw-syscall source for SIDs `100060`–
|
||||
`100066`, both with CO-RE parent lookup, shared cooldown/emission, and
|
||||
fail-open probe status. The syscall source also routes `setuid`/`setgid` into
|
||||
typed-host SIDs `100071`/`100072` and permission/ownership changes into SID
|
||||
`100070`, including relative-path resolution. Confirmed interpreter
|
||||
`write`/`writev`/`pwrite` variants,
|
||||
successful TCP connect/bind/listen/accept calls, `capset`, and `finit_module`
|
||||
now feed typed-host SIDs `100069`, `100067`/`100073`/`100068`/`100076`,
|
||||
`100077`, and `100078`. Phase 2 now also includes an opt-in hardened systemd
|
||||
validation unit, static build/install path, isolated baseline state, and
|
||||
journal JSONL. A bounded rotating JSONL copy provides isolated retained event
|
||||
history, and an atomic heartbeat gives external watchdogs a non-journal
|
||||
liveness signal. Bounded Python-schema-compatible JSON/text alert snapshots
|
||||
now provide truthful retained counts and basic process context. The Python
|
||||
service remains the production default until incident grouping, rich
|
||||
enrichment/assurance, management-console consumers, and broader subsystem
|
||||
parity land.
|
||||
|
||||
## Backlog
|
||||
|
||||
|
|
|
|||
|
|
@ -233,8 +233,26 @@ signature keeps `sid` / `signature` / `classtype` / tests / docs.
|
|||
`input_snooper`, `credential_access`, `stealth_network`, the
|
||||
`memory_obfuscation` cohort, and `egress`;
|
||||
Python remains the production agent.
|
||||
- **Phase 2 — eBPF via `cilium/ebpf`.** Port exec/syscall rules; drop the
|
||||
bcc/Python runtime dependency; single static binary.
|
||||
- **Phase 2 — eBPF via `cilium/ebpf`.** The transport-independent exec/syscall
|
||||
event models and rule engines have offline Python-envelope parity in Go. An
|
||||
opt-in native execve and filtered raw-syscall tracepoints now feed both rule
|
||||
engines through `cilium/ebpf` with CO-RE parent lookup and fail-open polling
|
||||
fallback; root UID/GID transitions and persistence-path permission changes
|
||||
plus capability-set and file-backed module-load requests also feed the
|
||||
typed-host engine. Interpreter `write`/`writev`/`pwrite` variants are
|
||||
enter/exit-correlated and filtered in-kernel before their FD paths are
|
||||
resolved in userspace. Successful
|
||||
connect/bind calls are likewise correlated, with TCP inode verification for
|
||||
connect events; listen/accept calls resolve both endpoints from the successful
|
||||
socket FD.
|
||||
An opt-in hardened systemd validation unit now runs the static Go binary with
|
||||
isolated baseline state, journal JSONL, a bounded retained JSONL stream, and
|
||||
an atomic health marker. All current typed-host families now have native
|
||||
transports. Bounded `enodia.alert.snapshot.v1` JSON/text pairs now retain
|
||||
alerts with basic process context and feed truthful status counts. Incident
|
||||
grouping, rich enrichment/assurance, management-consumer integration, and
|
||||
broader parity must land before the bcc/Python runtime dependency can be
|
||||
dropped.
|
||||
- **Phase 3 — Rule-engine parity + assimilation.** `rev`/`reference`/`metadata`,
|
||||
statebits (flowbits), thresholding, suppression, and — if rule count warrants
|
||||
— the MPM prefilter. `rules list/show/test/docs` parity.
|
||||
|
|
|
|||
|
|
@ -331,7 +331,7 @@ Store paths derive from `log_dir`: `listener-baseline.json`,
|
|||
|
||||
---
|
||||
|
||||
## 1.8 Port parity notes (Go 2.0)
|
||||
## 1.8 Port parity notes (Go port)
|
||||
|
||||
What the Go agent must reproduce to be behavior-equivalent for this subsystem:
|
||||
|
||||
|
|
@ -349,7 +349,8 @@ What the Go agent must reproduce to be behavior-equivalent for this subsystem:
|
|||
the other wrote.
|
||||
- **Reconcile-before-cooldown** ordering and 60s per-key cooldown (§1.6).
|
||||
|
||||
Current Go status (as of this writing): poll detectors are ported and pass the
|
||||
parity fixture, but baselines are still **injected from fixtures** — the store
|
||||
package, live SUID walk, and the daemon-equivalent lifecycle above are the
|
||||
work this document specifies.
|
||||
Current Go status: poll detectors pass the parity fixture. The baseline store,
|
||||
live SUID walk, startup grace, persistence scan, and first-seen retention are
|
||||
implemented behind the sidecar's explicit `--state-dir` lifecycle. Fixture mode
|
||||
continues to inject state and never touches a live store. Production service
|
||||
cutover remains outside this acceptance slice.
|
||||
|
|
|
|||
276
docs/behavior/02-poll-detectors.md
Normal file
276
docs/behavior/02-poll-detectors.md
Normal file
|
|
@ -0,0 +1,276 @@
|
|||
<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
|
||||
# 2. Poll Detectors
|
||||
|
||||
Scope: the per-detector behavioral contract for the twelve pure detectors in
|
||||
`detectors/`. Each is a `detect(state, cfg) -> Iterable[Alert]` over the
|
||||
`SystemState` built in §1. This section documents, per detector, the inputs it
|
||||
reads, the match algorithm, the exact alert it emits, and its allow-list /
|
||||
fail-open rules. The registry order and the baseline-arming gate are in §1.3 /
|
||||
§1.5; the alert fields and dedup keying are in §1.1.
|
||||
|
||||
Every detector is fail-open by inheritance: it reads already-captured
|
||||
`SystemState` fields, and missing `/proc`/`ss` data is an empty value, never an
|
||||
exception.
|
||||
|
||||
## 2.0 Signature summary
|
||||
|
||||
The stable contract. A port emitting a different SID, severity, classtype, or
|
||||
key for the same condition is a parity break.
|
||||
|
||||
| SID | signature | severity | classtype | key |
|
||||
|-----|-----------|----------|-----------|-----|
|
||||
| 100010 | `reverse_shell` | CRITICAL | `c2-reverse-shell` | `rsh:<pid>` |
|
||||
| 100011 | `ld_preload` | CRITICAL | `rootkit-preload` | `ldp:global`, `ldp:<pid>` |
|
||||
| 100012 | `deleted_exe` | CRITICAL | `fileless-execution` | `del:<pid>` |
|
||||
| 100013 | `new_listener` | HIGH | `backdoor-listener` | `lis:<port>/<comm>` |
|
||||
| 100014 | `new_suid` | CRITICAL / HIGH | `privilege-escalation` | `suid:<path>` |
|
||||
| 100015 | `persistence` | HIGH | `persistence` | `persist:<path>` |
|
||||
| 100016 | `egress` | HIGH | `c2-exfil` | `egr:<pid>:<host>` |
|
||||
| 100032 | `input_snooper` | HIGH | `credential-keylogging` | `input:<pid>:<target>` |
|
||||
| 100033 | `credential_access` | CRITICAL | `credential-theft` | `cred:<pid>:<target>` |
|
||||
| 100036 | `stealth_network` | MEDIUM / HIGH | `covert-channel` | `stealthnet:<kind>:<pid>:<local>:<peer>` |
|
||||
| 100039 | `memory_obfuscation` | HIGH / CRITICAL | `memory-obfuscation` | `mem:memory_obfuscation:<pid>:<lo>-<hi>` |
|
||||
| 100048 | `process_hiding_library` | CRITICAL | `process-hiding` | `mem:process_hiding_library:<pid>:<lo>-<hi>` |
|
||||
| 100049 | `process_injection_library` | HIGH | `process-injection` | `mem:process_injection_library:<pid>:<lo>-<hi>` |
|
||||
| 100074 | `first_public_destination` | MEDIUM | `network-rarity` | `firstdest:<comm>:<ip:port>` |
|
||||
| 100075 | `first_listener_port` | MEDIUM | `network-rarity` | `firstlisten:<comm>:<port>` |
|
||||
|
||||
`<lo>-<hi>` are the map bounds formatted as lowercase hex (`{start:x}-{end:x}`).
|
||||
|
||||
Allow-list config keys and their defaults live in `config.py`; this section
|
||||
names each key where used. The Go port mirrors the same defaults (see
|
||||
`go-agent/internal/config`), so the values are asserted by config tests, not
|
||||
repeated here.
|
||||
|
||||
---
|
||||
|
||||
## 2.1 Shared address helpers (`netutil.py`)
|
||||
|
||||
Used by `egress`, `stealth_network`, and `first_seen`.
|
||||
|
||||
- `split_host_port(endpoint)` — splits an `ss` endpoint. If it starts with `[`,
|
||||
partition on `]` (IPv6, e.g. `[::1]:22` → `::1`, `22`); else `rpartition(":")`
|
||||
(e.g. `10.0.0.2%eth0:443` → `10.0.0.2%eth0`, `443`). Returns `(host, port)`
|
||||
strings.
|
||||
- `parse_addr(addr)` — strips `[]`, drops an IPv6 `%zone`, returns an
|
||||
`ipaddress` object or `None`.
|
||||
- `is_public_ip(addr)` — `True` only if `parse_addr(...).is_global`: excludes
|
||||
loopback, RFC1918, link-local, CGNAT (100.64/10), and other non-global
|
||||
ranges. Unparseable → `False`.
|
||||
- `ip_in_cidrs(addr, cidrs)` — `True` if `addr` is inside any CIDR (the operator
|
||||
trust list); version-matched; bad CIDRs skipped.
|
||||
|
||||
## 2.2 Package provenance (`provenance.py`)
|
||||
|
||||
Used by `new_listener` (§2.11) as an optional suppressor.
|
||||
|
||||
`package_owner(path)` returns the owning package string or `None`, using the
|
||||
first available of `pacman -Qo` / `dpkg -S` / `rpm -qf` (5s timeout each, cached
|
||||
via `lru_cache`; `(deleted)` suffix stripped first). `is_package_owned(path) =
|
||||
package_owner(path) is not None`. **Contract:** ownership *ranks/explains*, it
|
||||
never proves safety — it may suppress a `new_listener` alert but is never used
|
||||
to silently drop anything else.
|
||||
|
||||
---
|
||||
|
||||
## 2.3 `reverse_shell` — SID 100010, CRITICAL
|
||||
|
||||
Interpreter process with a **network** socket on its stdio.
|
||||
|
||||
- Input: `state.processes`, `state.net_peer_by_inode`.
|
||||
- Match, per process: `proc.comm in cfg.interpreters`; `proc.stdio_socket_inode()`
|
||||
is not `None` (fd 0/1/2 links to `socket:[…]`); and that inode appears in the
|
||||
network socket table (`net_peer_by_inode`). The last condition is what
|
||||
excludes benign unix sockets / pipes and keeps false positives near zero.
|
||||
- Emit: key `rsh:<pid>`, `pids=(pid,)`, detail
|
||||
`pid=… comm=… stdio=net-socket peer=[local -> peer] cmd=[<cmdline first 90>]`.
|
||||
- Config: `interpreters` allow-*set of shell/scripting comms* (membership means
|
||||
"watch", not "trust").
|
||||
|
||||
## 2.4 `ld_preload` — SID 100011, CRITICAL
|
||||
|
||||
Two independent signatures, same SID:
|
||||
|
||||
1. **Global** — `/etc/ld.so.preload` exists and is non-empty (fail-open read).
|
||||
Key `ldp:global`, no pids, detail includes the file contents (newlines →
|
||||
spaces).
|
||||
2. **Per-process** — a process whose `environ["LD_PRELOAD"]` starts with a
|
||||
suspicious prefix: `("/tmp/", "/dev/shm/", "/var/tmp/", "/run/user/", "./")`.
|
||||
Key `ldp:<pid>`, `pids=(pid,)`.
|
||||
|
||||
Note the global check reads `/etc/ld.so.preload` directly (not the injected
|
||||
`state`), so a port must read it live inside the detector or supply it on state.
|
||||
|
||||
## 2.5 `deleted_exe` — SID 100012, CRITICAL
|
||||
|
||||
Process running from a deleted or memfd-backed binary in an attacker-controlled
|
||||
location.
|
||||
|
||||
- Match: `proc.exe` is fileless — `"memfd:" in exe`, **or** `"(deleted)" in exe`
|
||||
**and** `exe` starts with `("/tmp/", "/dev/shm/", "/var/tmp/", "/run/")`.
|
||||
Normal-path deleted exes (a daemon still running after a package upgrade) are
|
||||
ignored on purpose.
|
||||
- Emit: key `del:<pid>`, `pids=(pid,)`, detail `pid=… comm=… exe=[…]`.
|
||||
|
||||
## 2.6 `input_snooper` — SID 100032, HIGH
|
||||
|
||||
Process holding a keyboard/HID event device open (keylogger tell).
|
||||
|
||||
- Match, per process: skip if `comm in cfg.input_snooper_allow_comms`; then scan
|
||||
`fd_targets` for a target starting with `("/dev/input/event", "/dev/uinput",
|
||||
"/dev/hidraw")`. **First** matching fd emits and then `break` (at most one
|
||||
alert per process).
|
||||
- Emit: key `input:<pid>:<target>`, `pids=(pid,)`, detail
|
||||
`pid=… comm=… fd=… has input device open: <target>`.
|
||||
- Config: `input_snooper_allow_comms` (display servers, compositors, remappers).
|
||||
|
||||
## 2.7 `credential_access` — SID 100033, CRITICAL
|
||||
|
||||
Live process with a sensitive credential file open.
|
||||
|
||||
- Match, per process: skip if `comm in cfg.credential_access_allow_comms`; then
|
||||
per fd target (with a trailing ` (deleted)` stripped), classify via
|
||||
`_sensitive_kind`, first hit emits then `break`. Classification precedence:
|
||||
1. exact file in `{/etc/shadow, /etc/gshadow, /etc/security/opasswd}` →
|
||||
"system credential database";
|
||||
2. matches `cfg.credential_access_extra_paths` (entry ending `/` = prefix;
|
||||
else exact or `entry/` prefix) → "configured credential path";
|
||||
3. path contains `/.ssh/` and basename starts `id_` or ends `.pem`/`.key` →
|
||||
"private SSH key";
|
||||
4. path under `/etc/NetworkManager/system-connections/` → "NetworkManager
|
||||
secret profile";
|
||||
5. basename in `{logins.json, key4.db, "Login Data", Cookies}` **and** path
|
||||
(lowercased) contains a browser marker (`/.mozilla/`, `/firefox/`,
|
||||
`/chromium/`, `/google-chrome/`, `/chrome/`, `/brave`, `/vivaldi/`,
|
||||
`/edge/`) → "browser credential store".
|
||||
|
||||
Non-absolute paths are never sensitive.
|
||||
- Emit: key `cred:<pid>:<target>`, `pids=(pid,)`, detail
|
||||
`pid=… comm=… fd=… has <kind> open: <target>`.
|
||||
- Config: `credential_access_allow_comms`, `credential_access_extra_paths`.
|
||||
|
||||
## 2.8 `stealth_network` — SID 100036, MEDIUM or HIGH
|
||||
|
||||
Sockets in families attackers use to slide around TCP/UDP checks.
|
||||
|
||||
- Watched kinds: `{raw, sctp, dccp, packet, mptcp, tipc, xdp, vsock}` (matched
|
||||
on `sock.kind.lower()`).
|
||||
- Skip if: kind in `cfg.stealth_network_allow_kinds`; or `sock.comm` in
|
||||
`cfg.stealth_network_allow_comms`; or `sock.state` is non-empty and **not** in
|
||||
`{ESTAB, LISTEN, UNCONN, CONNECTED, SYN-SENT, SYN-RECV}`.
|
||||
- Severity: HIGH if kind in `{raw, packet, xdp}`; else HIGH if `state == LISTEN`
|
||||
or the peer is a public IP outside the egress trust list; else MEDIUM.
|
||||
- Emit: key `stealthnet:<kind>:<pid>:<local>:<peer>`, `pids=(pid,)` only when
|
||||
pid is known, detail `<kind> socket state=… local=… peer=… comm=<comm or ?>
|
||||
pid=<pid or ?>`.
|
||||
|
||||
## 2.9 `egress` — SID 100016, HIGH
|
||||
|
||||
Interpreter holding an outbound connection to a public IP.
|
||||
|
||||
- Match, per `established_sockets()`: `s.comm in cfg.interpreters`; peer host is
|
||||
public (`is_public_ip`); peer not in `cfg.egress_allow_cidrs`.
|
||||
- Emit: key `egr:<pid>:<host>`, `pids=(pid,)` when pid known, detail
|
||||
`pid=… comm=… -> <host>:<port> (interpreter to public IP)`.
|
||||
|
||||
## 2.10 `memory_obfuscation` — SIDs 100039 / 100048 / 100049
|
||||
|
||||
Inspects `/proc/<pid>/maps` shapes (never reads process memory). Per process,
|
||||
each map is classified by `_indicator`; **at most one alert per signature per
|
||||
process** (a per-process `seen` set). Precedence within a single map:
|
||||
|
||||
1. If `path` matches `cfg.memory_obfuscation_allow_paths` (prefix) → ignore.
|
||||
2. `path` basename contains a hide hint (`libhide`, `libprocesshide`,
|
||||
`process_hide`, `proc_hide`, `rootkit_hide`) → **`process_hiding_library`**,
|
||||
SID 100048, CRITICAL, classtype `process-hiding`.
|
||||
3. executable **and** library from a writable/runtime path (`/tmp/`, `/dev/shm/`,
|
||||
`/var/tmp/`, `/run/user/` with a `.so`/`.so.`/`.dylib` in the name) →
|
||||
**`process_injection_library`**, SID 100049, HIGH, classtype
|
||||
`process-injection`.
|
||||
4. executable **and** path contains `memfd:` or `(deleted)` →
|
||||
**`memory_obfuscation`**, SID 100039, CRITICAL.
|
||||
5. executable **and** writable (RWX) → `memory_obfuscation`, HIGH.
|
||||
6. executable **and** anonymous (`""`, `[heap]`, `[stack]`, `[anon…`,
|
||||
`[stack:…`) → `memory_obfuscation`, HIGH.
|
||||
|
||||
The `memory_obfuscation` signature (only) is additionally suppressed when
|
||||
`comm in cfg.memory_obfuscation_allow_comms` (JITs: java, node, browsers,
|
||||
dotnet, qemu, wine, wasmtime). The library signatures are **not** suppressed by
|
||||
that allow-list.
|
||||
|
||||
- Emit: key `mem:<signature>:<pid>:<lo>-<hi>`, `pids=(pid,)`, detail
|
||||
`pid=… comm=… <indicator detail> range=<lo>-<hi>`.
|
||||
|
||||
## 2.11 `new_listener` — SID 100013, HIGH
|
||||
|
||||
New listening socket absent from the startup baseline (see §1.5.1). No-op when
|
||||
`state.listener_baseline is None` (unarmed).
|
||||
|
||||
- Match, per `LISTEN` socket: identity `key = port/comm` where
|
||||
`port = local.rpartition(":")[2]` and `comm` defaults `?`. Skip if key in
|
||||
`state.listener_baseline`, or `port in cfg.listener_allow_ports`, or
|
||||
`comm in cfg.listener_allow_comms`.
|
||||
- Optional provenance gate: if `cfg.suppress_package_owned_listeners` and the
|
||||
owning process's `exe` `is_package_owned`, skip.
|
||||
- Emit: key `lis:<port>/<comm>`, `pids=(pid,)` when known, detail
|
||||
`new listening socket <local> by comm=<comm>`.
|
||||
|
||||
## 2.12 `new_suid` — SID 100014, HIGH or CRITICAL
|
||||
|
||||
SUID/SGID binary not in the baseline. No-op when `state.suid_binaries is None`
|
||||
**or** `state.suid_baseline is None` (see §1.5.3 — no alerts until an async scan
|
||||
result exists).
|
||||
|
||||
- Match, per path in `state.suid_binaries` not in `state.suid_baseline`:
|
||||
`hot = any(path.startswith(d.rstrip("/") + "/") for d in cfg.suid_hot_dirs)`.
|
||||
- Severity: CRITICAL if hot (writable dir), else HIGH.
|
||||
- Emit: key `suid:<path>`, no pids, detail
|
||||
`SUID/SGID binary in writable dir: <path>` (hot) or `new SUID/SGID binary:
|
||||
<path>`.
|
||||
|
||||
## 2.13 `persistence` — SID 100015, HIGH
|
||||
|
||||
Watched persistence-relevant file modified since the previous sweep. No-op when
|
||||
`state.persist_since is None` (unarmed).
|
||||
|
||||
- Input: walk `cfg.watch_persistence` (a dir → recurse `os.walk`, a file →
|
||||
itself). For each existing file, `mtime = os.lstat(path).st_mtime`
|
||||
(unreadable → skip).
|
||||
- Match: `mtime > state.persist_since` (the *previous* sweep's timestamp; §1.5.4).
|
||||
- Emit: key `persist:<path>`, no pids, detail `persistence file modified:
|
||||
<path>`.
|
||||
|
||||
## 2.14 `first_seen` — SIDs 100074 / 100075, MEDIUM
|
||||
|
||||
Network-rarity tracking with a self-owned persistent store. The store format,
|
||||
the `initialized` learn-once semantics, and the restart asymmetry are specified
|
||||
in §1.5.5 — not repeated here. Match summary:
|
||||
|
||||
- `first_public_destination` (100074): a `comm` reaches an `ESTAB` peer that is
|
||||
a public IP:port not previously recorded for it. Key
|
||||
`firstdest:<comm>:<ip:port>`, `pids=(pid,)` when known.
|
||||
- `first_listener_port` (100075): a `comm` opens a `LISTEN`/`UNCONN` tcp/udp
|
||||
socket on a numeric non-zero port not previously recorded for it. Key
|
||||
`firstlisten:<comm>:<port>`.
|
||||
|
||||
Both only alert when the store is already `initialized`; the recording of the
|
||||
new value happens regardless (so the alert fires once).
|
||||
|
||||
---
|
||||
|
||||
## 2.15 Port parity notes (Go port)
|
||||
|
||||
- The Go poll detectors for §2.3–§2.14 are **ported and pass the parity
|
||||
fixture** (`scripts/check-go-parity.py`), covering exact SID/severity/key/
|
||||
classtype/detail output. This document is the human-readable form of that
|
||||
fixture's contract.
|
||||
- `first_seen`/`new_listener`/`new_suid`/`persistence` receive injected baseline
|
||||
inputs in parity mode and use the opt-in live lifecycle described in §1 when
|
||||
`--state-dir` is supplied.
|
||||
- Watch two easily-missed rules when reviewing a port: the **first-match-then-
|
||||
break** behavior in `input_snooper`/`credential_access` (one alert per
|
||||
process), and the **per-signature `seen` dedup** in `memory_obfuscation` (one
|
||||
alert per signature per process, precedence-ordered).
|
||||
- `ld_preload`'s global check reads `/etc/ld.so.preload` **inside the detector**
|
||||
(or from an injected state field), not from the shared `SystemState` process
|
||||
view.
|
||||
147
docs/behavior/03-event-detection.md
Normal file
147
docs/behavior/03-event-detection.md
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
|
||||
# 3. Event Detection
|
||||
|
||||
Scope: the transport-independent contract for exec and syscall events and the
|
||||
rules that turn them into alerts. Python remains the oracle. The Go port
|
||||
accepts replay fixtures and reproduces the alert envelopes. Its opt-in native
|
||||
exec and syscall sources now feed the same contract through `cilium/ebpf`;
|
||||
typed-host kernel transports remain a separate boundary.
|
||||
|
||||
Typed host-event rules are part of the same replay and catalog contract.
|
||||
|
||||
## 3.1 Exec events
|
||||
|
||||
`ExecEvent` fields are `pid`, `ppid`, `uid`, `parent_comm`, `filename`, and
|
||||
`argv`. `argv` excludes `filename`.
|
||||
|
||||
- `exec_comm` is the POSIX basename of `filename`. A trailing slash therefore
|
||||
produces an empty basename; ports must not clean the path first.
|
||||
- `argv_str` is `filename` and every captured argv item joined with one space,
|
||||
then stripped at both ends.
|
||||
- Rules AND every condition they specify. Values within one condition are ORed.
|
||||
- A rule with no positive condition is invalid. `parent_exclude` is a veto, not
|
||||
a positive condition.
|
||||
- `argv_regex` is case-insensitive and searches the whole `argv_str`.
|
||||
|
||||
Alert contract:
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| signature | `exec_rule.<classtype>` |
|
||||
| key | `exec:<sid>:<pid>` |
|
||||
| pids | the event PID, even when zero |
|
||||
| detail | `sid=<sid> <msg> — pid=… ppid=… parent=… exec=… argv=[…]` |
|
||||
|
||||
The argv text in detail is truncated to 120 characters. Built-ins:
|
||||
|
||||
| SID | Severity | Classtype | Match |
|
||||
|---|---|---|---|
|
||||
| 100001 | CRITICAL | `fileless-execution` | filename begins `/tmp/`, `/dev/shm/`, or `/var/tmp/` |
|
||||
| 100002 | CRITICAL | `c2-reverse-shell` | reverse-shell regex in `argv_str` |
|
||||
| 100003 | CRITICAL | `web-rce` | web/DB parent launches a configured interpreter basename |
|
||||
| 100004 | HIGH | `ingress-tool-transfer` | curl/wget/fetch piped to a shell |
|
||||
|
||||
Configured rules are appended after built-ins from `[[exec_rules]]` tables in
|
||||
`exec_rules_file`. Missing files mean built-ins only. Unknown severity strings
|
||||
fall back to HIGH, matching Python.
|
||||
|
||||
## 3.2 Syscall events
|
||||
|
||||
`SyscallEvent` fields are `pid`, `ppid`, `uid`, `comm`, `syscall`, six unsigned
|
||||
arguments (`arg0`…`arg5`), and optional `text`. Missing arguments are zero.
|
||||
|
||||
Alert contract:
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| signature | `syscall_rule.<classtype>` |
|
||||
| key | `syscall:<sid>:<pid>:<syscall>:<arg0-hex>:<arg2-hex>` |
|
||||
| pids | the event PID |
|
||||
| detail | rule metadata plus `arg0`…`arg3` formatted as prefixed lowercase hex |
|
||||
|
||||
Optional text is appended as ` text=[…]` and truncated to 80 characters.
|
||||
|
||||
| SID | Severity | Classtype | Match |
|
||||
|---|---|---|---|
|
||||
| 100060 | CRITICAL | `memory-obfuscation` | `mprotect`, arg2 contains both WRITE (`0x2`) and EXEC (`0x4`) |
|
||||
| 100061 | CRITICAL | `memory-obfuscation` | `mmap`, arg2 contains WRITE and EXEC |
|
||||
| 100062 | MEDIUM | `fileless-execution` | any `memfd_create` |
|
||||
| 100063 | HIGH | `anti-analysis` | `ptrace` request TRACEME (`0`), ATTACH (`16`), or SEIZE (`0x4206`) |
|
||||
| 100064 | MEDIUM | `anti-analysis` | any `seccomp`, or `prctl` arg0 `PR_SET_SECCOMP` (`22`) |
|
||||
| 100065 | HIGH | `credential-access` | `process_vm_readv` or `process_vm_writev` |
|
||||
| 100066 | MEDIUM | `memory-obfuscation` | `mlock`, `mlock2`, or `mlockall` |
|
||||
|
||||
## 3.3 Current Go acceptance boundary
|
||||
|
||||
`scripts/check-go-parity.py` replays shared JSON into both implementations and
|
||||
compares decoded `enodia.event.v1` alert envelopes exactly. Acceptance requires:
|
||||
|
||||
- all four built-in exec rules plus one configured rule;
|
||||
- all seven syscall rules and all ten host rules plus negative predicate cases;
|
||||
- identical SID, severity, signature, classtype, key, detail, PID list, host,
|
||||
and timestamp;
|
||||
- no BPF/root/kernel dependency for replay mode.
|
||||
|
||||
Replay flags are `--exec-events`, `--syscall-events`, and `--host-events`.
|
||||
They emit alert JSONL only and exit. They remain deterministic verification
|
||||
surfaces independent of the live source.
|
||||
|
||||
`--ebpf-exec` loads a compiled tracepoint program with `cilium/ebpf`, captures
|
||||
PID, parent PID, UID, caller comm, filename, and the first two arguments, then
|
||||
passes each record through the same exec rule engine, shared cooldown gate, and
|
||||
`enodia.event.v1` builder. Parent lookup uses CO-RE relocation against kernel
|
||||
BTF. Startup and reader failures update `ebpf`/`ebpf_exec` status and leave the
|
||||
poll loop running. The flag is opt-in and requires Go 1.25+ to build.
|
||||
|
||||
`--ebpf-syscall` attaches one raw-syscall entry tracepoint. An architecture-
|
||||
specific map allows only `mprotect`, `mmap`, `memfd_create`, `ptrace`, `prctl`,
|
||||
`seccomp`, `process_vm_readv`, `process_vm_writev`, `mlock`, `mlock2`, and
|
||||
`mlockall` through the BPF program, plus `setuid` and `setgid`, avoiding a
|
||||
user-space event for every host syscall. It captures the six arguments and the
|
||||
memfd name before routing syscall events through SIDs `100060`–`100066`.
|
||||
`setuid`/`setgid` become typed host events for SIDs `100071`/`100072`. The
|
||||
portable `fchmodat`/`fchownat` calls, plus legacy `chmod`/`chown`/`lchown` on
|
||||
amd64, become `chmod`/`chown` host events for SID `100070`. Relative paths are
|
||||
resolved against `/proc/<pid>/cwd` before matching persistence prefixes. The
|
||||
`capset` effective bitmap is decoded to canonical capability names for SID
|
||||
`100077`. `finit_module` resolves its file descriptor through `/proc/<pid>/fd`
|
||||
and supplies path/module fields to SID `100078`. Syscall numbers come from
|
||||
build-target `golang.org/x/sys/unix` constants rather
|
||||
than a hand-maintained architecture table. Static builds are exercised for
|
||||
`amd64`, `arm64`, `riscv64`, and big-endian `s390x`.
|
||||
|
||||
## 3.4 Typed host events
|
||||
|
||||
`HostEvent` covers `tcp_connect`, `listen`, `bind`, `accept`, `file_write`,
|
||||
`chmod`, `chown`, `setuid`, `setgid`, `capset`, and `module_load`. Compatibility
|
||||
aliases accepted by `ruleops` are stable: `type`, `remote_*`, `bind_*`,
|
||||
`listen_*`, `uid_target`, `gid_target`, `caps`, `cap_effective`, singular
|
||||
`capability`, and module `name`. Numeric strings accept decimal/hex/octal;
|
||||
absent target UID/GID values are `-1`, not zero.
|
||||
|
||||
Host alerts use signature `host_rule.<classtype>` and a key containing every
|
||||
typed identity field. Capabilities are normalized uppercase for matching and
|
||||
rendering. Built-in SIDs are `100067`–`100073` and `100076`–`100078`.
|
||||
|
||||
## 3.5 Mixed pipeline and catalog
|
||||
|
||||
`--event-stream` accepts mixed JSONL (or `-` for stdin), infers exec/syscall
|
||||
events when no explicit type exists, and routes typed events through one
|
||||
transport-independent rule set. `--rules-list` and `--rules-show` expose the
|
||||
same sorted metadata and conditions as Python `ruleops`.
|
||||
|
||||
The native exec/syscall readers and all replay modes feed this boundary. The
|
||||
syscall reader also supplies typed-host transports for root UID/GID transition
|
||||
requests, persistence-path permission/ownership changes, and capability-set
|
||||
requests. `finit_module` supplies module-load events with resolved paths.
|
||||
Interpreter `write`, `writev`, `pwrite64`, `pwritev`, and `pwritev2` calls are
|
||||
filtered by the same comm list as SID `100069`, correlated across raw syscall
|
||||
enter/exit in a bounded BPF map, emitted only for positive byte counts, and
|
||||
resolved through `/proc/<pid>/fd`. Successful
|
||||
IPv4/IPv6 `connect` and `bind` calls use the same enter/exit correlation and
|
||||
in-kernel interpreter filter for SIDs `100067`/`100073`. Connect events are
|
||||
verified against `/proc/net/tcp{,6}` by socket inode so UDP cannot be mislabeled
|
||||
as `tcp_connect`. Successful `listen`, `accept`, and `accept4` calls resolve
|
||||
their socket FD to local/peer TCP endpoints through the same inode tables for
|
||||
SIDs `100068`/`100076`. Every current typed-host event family now has a native
|
||||
transport.
|
||||
|
|
@ -38,8 +38,8 @@ Written in port-priority order. Status reflects which documents exist.
|
|||
| # | File | Covers | Status |
|
||||
|---|------|--------|--------|
|
||||
| 1 | [`01-sweep-and-state.md`](01-sweep-and-state.md) | `system.py` capture, `daemon.py` sweep loop + baseline lifecycle, `alert.py`, detector registry, `config.py` timing knobs | **written** |
|
||||
| 2 | `02-poll-detectors.md` | the twelve `detectors/*` — per-detector input → match → output contract | planned |
|
||||
| 3 | `03-event-detection.md` | `event.py`, exec/syscall/host rule engines, `ruleops.py`, `sids.py` | planned |
|
||||
| 2 | [`02-poll-detectors.md`](02-poll-detectors.md) | the twelve `detectors/*` — per-detector input → match → output contract | **written** |
|
||||
| 3 | [`03-event-detection.md`](03-event-detection.md) | `event.py`, exec/syscall/host rule engines, `ruleops.py`, `sids.py` | **written** |
|
||||
| 4 | `04-integrity.md` | `fim.py`, `pkgdb.py`, `selfprotect.py`, `assurance.py` | planned |
|
||||
| 5 | `05-rootcheck.md` | `rootcheck.py` anti-rootkit cross-view checks | planned |
|
||||
| 6 | `06-posture.md` | `posture.py` advisory host-hygiene checks | planned |
|
||||
|
|
|
|||
295
docs/superpowers/plans/2026-07-10-eve-event-envelope.md
Normal file
295
docs/superpowers/plans/2026-07-10-eve-event-envelope.md
Normal file
|
|
@ -0,0 +1,295 @@
|
|||
# EVE-Style Event Envelope (Phase 0) Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add an additive, EVE-style event envelope (`enodia.event.v1`) — a single typed wrapper with an `event_type` discriminator — as the reference contract that both today's Python and a future Go agent will emit.
|
||||
|
||||
**Architecture:** A new pure `enodia_sentinel/event.py` module provides a reference encoder that wraps any payload in a common envelope (`schema`, `event_type`, `timestamp`, `host`, and a payload sub-object keyed by the event type — exactly like Suricata EVE). It is *additive*: it does not change any existing emitter or break existing v1 contracts. Existing objects (starting with `Alert`) get a thin convenience wrapper. Compatibility tests and `SCHEMAS.md` pin the contract.
|
||||
|
||||
**Tech Stack:** Python 3 standard library only (`dataclasses`/functions, `datetime`, `os`), `unittest`.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **Stdlib-only.** No third-party imports in `enodia_sentinel/`. (Copied from CLAUDE.md engineering rules.)
|
||||
- **Additive v1.** Required v1 fields are never removed, renamed, or retyped; new work only adds. (Copied from `docs/SCHEMAS.md`.)
|
||||
- **Every operator-facing JSON contract updates `docs/SCHEMAS.md` and compatibility tests.** (Copied from CLAUDE.md documentation checklist.)
|
||||
- **SPDX header** `# SPDX-License-Identifier: GPL-3.0-or-later` as the first line of every new `.py` file. (Matches every existing module.)
|
||||
- **Shared checkout:** stage only files this plan creates/modifies; never `git add -A`. (Copied from CLAUDE.md.)
|
||||
- **Timestamp format:** `datetime.now().astimezone().isoformat()`. **Host:** `os.uname().nodename`. (Existing convention in `snapshot.py`.)
|
||||
- **Test command:** `python3 -m unittest <module> -v` from the repo root.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Event envelope reference encoder
|
||||
|
||||
**Files:**
|
||||
- Modify: `enodia_sentinel/schemas.py` (add one constant)
|
||||
- Create: `enodia_sentinel/event.py`
|
||||
- Create: `tests/test_event_envelope.py`
|
||||
- Modify: `docs/SCHEMAS.md` (add `enodia.event.v1` section)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: nothing (leaf module).
|
||||
- Produces:
|
||||
- `enodia_sentinel.schemas.EVENT_V1: str == "enodia.event.v1"`
|
||||
- `enodia_sentinel.event.EVENT_TYPES: frozenset[str]` — the closed v1 set `{"alert", "incident", "status"}`
|
||||
- `enodia_sentinel.event.build_event(event_type: str, payload: dict, *, host: str | None = None, timestamp: str | None = None) -> dict` — returns an envelope dict with keys `schema`, `event_type`, `timestamp`, `host`, and `event_type`'s value used as a key holding `payload`. Raises `ValueError` for an `event_type` not in `EVENT_TYPES`.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Create `tests/test_event_envelope.py`:
|
||||
|
||||
```python
|
||||
# SPDX-License-Identifier: GPL-3.0-or-later
|
||||
"""Compatibility tests for the enodia.event.v1 envelope."""
|
||||
import os
|
||||
import unittest
|
||||
from datetime import datetime
|
||||
|
||||
from enodia_sentinel import event, schemas
|
||||
|
||||
|
||||
class TestEventEnvelope(unittest.TestCase):
|
||||
def test_schema_id_is_stable(self):
|
||||
self.assertEqual(schemas.EVENT_V1, "enodia.event.v1")
|
||||
|
||||
def test_build_event_shape_and_types(self):
|
||||
env = event.build_event("alert", {"sid": 100010})
|
||||
self.assertEqual(env["schema"], schemas.EVENT_V1)
|
||||
self.assertEqual(env["event_type"], "alert")
|
||||
self.assertIsInstance(env["timestamp"], str)
|
||||
self.assertIsInstance(env["host"], str)
|
||||
# EVE-style: payload lives under a key named by event_type.
|
||||
self.assertEqual(env["alert"], {"sid": 100010})
|
||||
|
||||
def test_timestamp_defaults_to_iso8601(self):
|
||||
env = event.build_event("status", {"running": True})
|
||||
# Parses as ISO-8601 without raising.
|
||||
datetime.fromisoformat(env["timestamp"])
|
||||
|
||||
def test_host_defaults_to_nodename(self):
|
||||
env = event.build_event("status", {"running": True})
|
||||
self.assertEqual(env["host"], os.uname().nodename)
|
||||
|
||||
def test_overrides_are_respected(self):
|
||||
env = event.build_event(
|
||||
"incident", {"id": "x"}, host="host-a", timestamp="2026-07-10T00:00:00-07:00")
|
||||
self.assertEqual(env["host"], "host-a")
|
||||
self.assertEqual(env["timestamp"], "2026-07-10T00:00:00-07:00")
|
||||
|
||||
def test_unknown_event_type_rejected(self):
|
||||
with self.assertRaises(ValueError):
|
||||
event.build_event("nope", {})
|
||||
|
||||
def test_event_types_is_closed_set(self):
|
||||
self.assertEqual(event.EVENT_TYPES, frozenset({"alert", "incident", "status"}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `python3 -m unittest tests.test_event_envelope -v`
|
||||
Expected: FAIL — `ModuleNotFoundError: No module named 'enodia_sentinel.event'` (and `AttributeError` on `schemas.EVENT_V1`).
|
||||
|
||||
- [ ] **Step 3: Add the schema constant**
|
||||
|
||||
In `enodia_sentinel/schemas.py`, after the `RECONCILE_V1` line (currently the last constant), add:
|
||||
|
||||
```python
|
||||
EVENT_V1 = "enodia.event.v1"
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Write the module**
|
||||
|
||||
Create `enodia_sentinel/event.py`:
|
||||
|
||||
```python
|
||||
# SPDX-License-Identifier: GPL-3.0-or-later
|
||||
"""EVE-style event envelope: one typed wrapper for every emitted record.
|
||||
|
||||
Every envelope carries a stable ``schema`` id, an ``event_type`` discriminator,
|
||||
a ``timestamp`` and ``host``, and the type-specific payload under a key named by
|
||||
the event type (mirroring Suricata's EVE JSON, where an ``alert`` record holds
|
||||
its detail under ``"alert"``). This is the reference contract a future Go agent
|
||||
must emit byte-for-byte; the Python side is the oracle.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from datetime import datetime
|
||||
|
||||
from .schemas import EVENT_V1
|
||||
|
||||
# Closed set of v1 discriminators. New types are added additively in later
|
||||
# phases (e.g. "dns", "tls", "http", "flow", "anomaly", "capture"); removing or
|
||||
# renaming a value is a breaking change.
|
||||
EVENT_TYPES = frozenset({"alert", "incident", "status"})
|
||||
|
||||
|
||||
def build_event(
|
||||
event_type: str,
|
||||
payload: dict,
|
||||
*,
|
||||
host: str | None = None,
|
||||
timestamp: str | None = None,
|
||||
) -> dict:
|
||||
"""Wrap ``payload`` in an ``enodia.event.v1`` envelope.
|
||||
|
||||
Raises ``ValueError`` if ``event_type`` is not a known v1 discriminator.
|
||||
"""
|
||||
if event_type not in EVENT_TYPES:
|
||||
raise ValueError(f"unknown event_type: {event_type!r}")
|
||||
if host is None:
|
||||
host = os.uname().nodename
|
||||
if timestamp is None:
|
||||
timestamp = datetime.now().astimezone().isoformat()
|
||||
return {
|
||||
"schema": EVENT_V1,
|
||||
"event_type": event_type,
|
||||
"timestamp": timestamp,
|
||||
"host": host,
|
||||
event_type: payload,
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run test to verify it passes**
|
||||
|
||||
Run: `python3 -m unittest tests.test_event_envelope -v`
|
||||
Expected: PASS (6 tests).
|
||||
|
||||
- [ ] **Step 6: Document the contract**
|
||||
|
||||
In `docs/SCHEMAS.md`, add this section immediately after the intro paragraph that ends "...pin the required v1 fields." (before the `## enodia.alert.v1` heading):
|
||||
|
||||
```markdown
|
||||
## `enodia.event.v1`
|
||||
|
||||
Uniform envelope that wraps every emitted record with a typed discriminator.
|
||||
This is the Python↔Go interoperability contract: any implementation must emit
|
||||
the same envelope for the same input. Modeled on Suricata's EVE JSON — the
|
||||
type-specific payload lives under a key named by `event_type`.
|
||||
|
||||
Required fields:
|
||||
|
||||
| Field | Type | Meaning |
|
||||
|---|---|---|
|
||||
| `schema` | string | `enodia.event.v1`. |
|
||||
| `event_type` | string | Discriminator: `alert`, `incident`, or `status` (additive set). |
|
||||
| `timestamp` | string | ISO-8601 emission time. |
|
||||
| `host` | string | Hostname at emission time. |
|
||||
| `<event_type>` | object | Payload, under a key equal to `event_type` (e.g. `alert` holds an `enodia.alert.v1` object). |
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add enodia_sentinel/schemas.py enodia_sentinel/event.py tests/test_event_envelope.py docs/SCHEMAS.md
|
||||
git commit -m "feat(schema): add enodia.event.v1 EVE-style event envelope"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Alert → event convenience wrapper
|
||||
|
||||
**Files:**
|
||||
- Modify: `enodia_sentinel/event.py`
|
||||
- Modify: `tests/test_event_envelope.py`
|
||||
- Modify: `docs/SCHEMAS.md` (one clarifying line)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `event.build_event` (Task 1); `enodia_sentinel.alert.Alert` with its existing `.to_dict()` method returning keys `sid, severity, signature, classtype, key, detail, pids`.
|
||||
- Produces: `enodia_sentinel.event.from_alert(alert, *, host: str | None = None, timestamp: str | None = None) -> dict` — an `enodia.event.v1` envelope with `event_type == "alert"` whose `alert` payload is exactly `alert.to_dict()`.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Append to `tests/test_event_envelope.py` inside `TestEventEnvelope` (before the `if __name__` guard):
|
||||
|
||||
```python
|
||||
def test_from_alert_wraps_alert_payload(self):
|
||||
from enodia_sentinel.alert import Alert, Severity
|
||||
alert = Alert(
|
||||
Severity.CRITICAL, "reverse_shell", "rsh:4242",
|
||||
"pid=4242 peer=[8.8.8.8:4444]", (4242,),
|
||||
sid=100010, classtype="command-and-control")
|
||||
env = event.from_alert(alert, host="host-a")
|
||||
self.assertEqual(env["event_type"], "alert")
|
||||
self.assertEqual(env["host"], "host-a")
|
||||
# Payload is the untouched enodia.alert.v1 object.
|
||||
self.assertEqual(env["alert"], alert.to_dict())
|
||||
self.assertEqual(env["alert"]["signature"], "reverse_shell")
|
||||
self.assertEqual(env["alert"]["sid"], 100010)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `python3 -m unittest tests.test_event_envelope.TestEventEnvelope.test_from_alert_wraps_alert_payload -v`
|
||||
Expected: FAIL — `AttributeError: module 'enodia_sentinel.event' has no attribute 'from_alert'`.
|
||||
|
||||
- [ ] **Step 3: Add the wrapper**
|
||||
|
||||
Append to `enodia_sentinel/event.py` (after `build_event`):
|
||||
|
||||
```python
|
||||
def from_alert(alert, *, host: str | None = None, timestamp: str | None = None) -> dict:
|
||||
"""Wrap an ``Alert`` (via its ``to_dict()``) in an ``alert`` event envelope."""
|
||||
return build_event("alert", alert.to_dict(), host=host, timestamp=timestamp)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `python3 -m unittest tests.test_event_envelope -v`
|
||||
Expected: PASS (7 tests).
|
||||
|
||||
- [ ] **Step 5: Document the alert mapping**
|
||||
|
||||
In `docs/SCHEMAS.md`, in the `enodia.event.v1` table row for `<event_type>`, the text already notes `alert` holds an `enodia.alert.v1` object. Add one line directly beneath the table:
|
||||
|
||||
```markdown
|
||||
The reference encoder lives in `enodia_sentinel/event.py`
|
||||
(`build_event`, `from_alert`); `tests/test_event_envelope.py` pins the contract.
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add enodia_sentinel/event.py tests/test_event_envelope.py docs/SCHEMAS.md
|
||||
git commit -m "feat(schema): add Alert -> enodia.event.v1 reference wrapper"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Full-suite regression check
|
||||
|
||||
**Files:** none (verification only).
|
||||
|
||||
**Interfaces:** none.
|
||||
|
||||
- [ ] **Step 1: Run the whole suite to confirm nothing regressed**
|
||||
|
||||
Run: `python3 -m unittest discover -s tests -v`
|
||||
Expected: PASS — the existing suite plus the 7 new envelope tests. No existing schema-contract test changes behavior (the work is purely additive).
|
||||
|
||||
- [ ] **Step 2: Confirm the additive guarantee held**
|
||||
|
||||
Run: `git diff --stat HEAD~2 -- enodia_sentinel/`
|
||||
Expected: only `schemas.py` (one added line) and the new `event.py` appear — no existing emitter (`alert.py`, `snapshot.py`, `incident.py`, `web.py`) was modified, proving the change is non-breaking.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
**Spec coverage (Phase 0 = "Freeze and extend the v1 EVE-style event envelope in SCHEMAS.md; `event_type` discriminator; alert/incident/status as event types; the Python↔Go contract; no language change yet"):**
|
||||
- EVE-style envelope with `event_type` discriminator → Task 1 (`build_event`, `EVENT_TYPES`).
|
||||
- alert/incident/status as event types → Task 1 (`EVENT_TYPES = {"alert","incident","status"}`); alert reference mapping → Task 2 (`from_alert`). Incident/status wrappers are deferred to their emitter phases; the discriminator set already reserves them additively, satisfying "as event types" at the contract level.
|
||||
- Documented in `SCHEMAS.md` → Tasks 1 & 2.
|
||||
- Compatibility tests pin the contract → Tasks 1 & 2; regression proven → Task 3.
|
||||
- No language change → confirmed; all Python, additive.
|
||||
|
||||
**Placeholder scan:** none — every code and doc step shows exact content.
|
||||
|
||||
**Type consistency:** `build_event(event_type, payload, *, host, timestamp)` and `from_alert(alert, *, host, timestamp)` signatures match between the interface blocks, the code steps, and the tests. `EVENT_V1`/`EVENT_TYPES` names are consistent across schemas.py, event.py, and tests. Envelope keys (`schema`, `event_type`, `timestamp`, `host`, `<event_type>`) are identical in code, tests, and the SCHEMAS.md table.
|
||||
|
||||
**Scope:** single subsystem, additive, independently testable; no decomposition needed.
|
||||
Loading…
Add table
Add a link
Reference in a new issue