enodia-sentinal/docs/GO_PORT_HANDOFF.md

# Go Port Handoff

Saved: 2026-07-22T04:08:40-07:00
Updated: 2026-07-24T04:45:55-07:00
Branch: `main`
Base commit: `a25c73a` (`feat(go): report integrity anchor context`); the FIM/
package-DB/rootcheck engines and the Python go-sidecar dashboard consumer
described below landed on top of that commit and remain uncommitted.
Status: items 1 and 2 of "Resume here" are now implemented and green; see the
updated sections below. Desktop GUI work referenced by the original checkpoint
has since been committed (`457c760`, `1d1dee7`, and related commits) and is no
longer part of this checkout's uncommitted state.

## Worktree warning

The checkout is intentionally dirty and contains work from multiple related
continuations. Do not reset, clean, or broadly restage it.

- The validation-sidecar, incident-persistence, bounded-enrichment, local-
  assurance, asynchronous-enrichment, incident-reader, package-ownership, and
  integrity-anchor tranches are signed commits `f85c2e8`, `538d1d9`,
  `fd2bbba`, `1ab4add`, `eff31b3`, `6355fb4`, `88d7a6e`, and `a25c73a`.
- On top of that, uncommitted work now adds `go-agent/internal/{fim,pkgdb,
  rootcheck}/`, wires them into `agent.Sweep`/`Initialize` behind
  `--state-dir`, and adds Python's read-only `/api/go-sidecar/*` consumer
  (`enodia_sentinel/web.py`, `schemas.GO_SIDECAR_V1`) plus a Sidecar dashboard
  tab. `git status --short` currently shows this as modified files across both
  trees plus the three new untracked Go package directories.
- `build/` is ignored and contains only local build output.

## Implemented Go surface

The Go module requires Go 1.25 or newer and is currently verified with
`go1.26.5-X:nodwarf5`. It uses `github.com/cilium/ebpf` v0.22.0 and builds as a
static binary.

### Polling and lifecycle

- All current Python poll detectors have Go parity: `reverse_shell`,
  `ld_preload`, `deleted_exe`, `input_snooper`, `credential_access`,
  `stealth_network`, `memory_obfuscation`, `egress`, `first_seen`,
  `new_listener`, `persistence`, and `new_suid`.
- `--state-dir` enables isolated listener/SUID baselines, startup grace,
  persistence scans, and atomic first-seen retention.
- The Go state directory must never be the Python daemon's
  `/var/log/enodia-sentinel` directory.

### Event rules and native sources

- Exec rules: built-in SIDs `100001`-`100004` plus configured rules.
- Syscall rules: SIDs `100060`-`100066`.
- Typed host rules: SIDs `100067`-`100073` and `100076`-`100078`.
- Mixed JSONL routing, rule catalog operations, correlation SID `100080`, and
  one shared cooldown gate are implemented.
- Embedded CO-RE exec and filtered raw-syscall probes use `cilium/ebpf` and
  fail open to polling on load/attach/read failures.
- Native typed transports include UID/GID changes, chmod/chown variants,
  `capset`, `finit_module`, confirmed interpreter `write`/`writev`/`pwrite64`/
  `pwritev`/`pwritev2`, and successful TCP connect/bind/listen/accept calls
  with `/proc` endpoint resolution.
- Generated little- and big-endian BPF objects live under
  `go-agent/internal/ebpfsource/`. Run `make generate-go` only after changing
  `go-agent/internal/ebpfsource/bpf/*.bpf.c`.

### Production validation path

- `make build-go` creates a stripped, static, VCS-independent binary.
- `make install-go` installs the separate binary and
  `enodia-sentinel-go-sidecar.service` without replacing or enabling the
  Python service.
- The hardened systemd unit has a current `systemd-analyze security` exposure
  score of 4.3 (`OK`). It preserves the host `/proc`, `/tmp`, and network view
  because detectors require them, while hiding the Python state tree.
- The unit uses `Type=notify`; READY is sent only after baseline setup, event
  log preflight, and both requested probe-load attempts.
- JSONL remains visible in journald and is also retained as private bounded
  `/var/lib/enodia-sentinel-go/events.jsonl` plus `events.jsonl.1` (64 MiB per
  active segment boundary). Non-status events are synchronized to disk.
- `/var/lib/enodia-sentinel-go/heartbeat` is atomically refreshed after each
  successful status emission.
- `enodia-sentinel-go --health` returns `enodia.go.health.v1` JSON and a
  watchdog-friendly exit status.
- `enodia-sentinel-go --events-tail N` reads active and rotated retained events
  chronologically, validates JSON, fails closed on corruption, and deliberately
  works without loading the agent configuration.
- The unit now retains private `alert-YYYYMMDD-HHMMSS.{json,log}` pairs using
  the required `enodia.alert.snapshot.v1` fields. Same-second alerts merge into
  one pair, process context is collected best-effort from `/proc`, and shared
  `max_snapshots` / `max_snapshot_age_days` limits are applied at startup,
  capture, and status time.
- `enodia.status.v1` totals, severity counts, and last-alert time are populated
  only from parseable retained Go snapshots. Python's unchanged `list_alerts`
  and `get_alert` data layer successfully consumed a Go-produced pair.
- Snapshot incident IDs now reference an atomic private `incidents.json` using
  the required `enodia.incident.v1` fields. Grouping mirrors Python's process-
  lineage-first, time-window fallback behavior, the index is bounded to 1000
  recent incidents, and same-second alert batches update one incident without
  duplicating the snapshot name.
- Incident severity/signatures/SIDs/PIDs/lineage/timestamps are updated
  additively and correlation SID `100080` is persisted. Python's unchanged
  `list_incidents` and `get_incident` consumers successfully loaded a live
  Go-produced index, snapshot, and timeline.
- A bounded no-I/O enrichment stage now derives Python-shaped process/parent
  context, lineage, remote-IP classification, and candidate paths entirely from
  already-captured records. Alert/process/indicator caps bound attacker-
  controlled work on the emission path.
- Snapshot log/JSON revisions now append synchronized private
  `enodia.hash_chain.v1` records. Artifact SHA-256 is streamed, previous-hash
  lookup reads a bounded chain tail, and append completion is synced before the
  alert event is emitted.
- A single 16-job asynchronous worker now streams SHA-256 only for regular
  executables at most 8 MiB and adds lstat file metadata. Slow I/O occurs
  outside the snapshot lock; a full queue simply leaves optional fields unknown.
- The asynchronous worker now resolves pacman/dpkg/rpm package-owner strings
  with a two-second bound and cache; ownership lookup never holds a detector
  lock or blocks alert capture.
- The worker reports FIM-baseline, package-DB-anchor, and Go hash-chain
  presence/age, plus the configured `pacman -Qkk` and rootcheck cadences now
  that those engines are ported (see below). This remains metadata only:
  recording a snapshot never starts integrity work itself.
- Notification fan-out is not yet ported.
- `--incidents-list` and `--incident-show <id>` now read the sidecar's isolated
  state without starting the agent. The latter returns `enodia.incident.view.v1`
  with the incident, available snapshots, and time-ordered timeline.
- No live system service was installed or enabled during development.

### FIM, package-DB anchor, and rootcheck engines (new since the base commit)

- `go-agent/internal/fim` is a sidecar-only engine, gated on `--state-dir`,
  that persists a SHA-256 security-path baseline as `fim-baseline.json` and
  performs non-overlapping background comparisons on `fim_scan_interval`,
  emitting the Python-identical SIDs `100017`-`100019`. `fim_pkg_verify = true`
  opts into a bounded `pacman -Qkk` verification pass emitting SID `100020`.
- `go-agent/internal/pkgdb` fingerprints pacman's local checksum database on
  `pkgdb_interval` and alerts (SID `100021`) only on changes lacking a
  corresponding transaction-log record; it never refreshes an unexplained
  changed anchor.
- `go-agent/internal/rootcheck` bounds PID cross-view probing with
  `rootcheck_pid_cap`, runs one background comparison per
  `rootcheck_interval`, and reproduces the Python rootcheck SIDs (hidden
  process/module/port families, promiscuous mode, module/kernel taint) from
  procfs, sysfs, `ps`, and socket views.
- All three run asynchronously off the sweep loop (`agent.Sweep` calls
  `MaybeStart` before detector evaluation and `Drain` after, so a slow hash
  tree or `pacman` invocation never delays a sweep) and return alerts through
  the existing cooldown/snapshot/incident pipeline rather than emitting
  directly.
- `enodia_sentinel/web.py` adds a read-only, schema-checked consumer for this
  evidence: `go_sidecar_state_dir` (config key) exposes `/api/go-sidecar/
  {status,alerts,incidents,events}` and a dashboard Sidecar tab. It never
  starts, stops, or writes into the Go sidecar's state directory, and a
  corrupted retained event log fails the request closed rather than showing a
  partial stream. The new `enodia.go.sidecar.v1` schema lives in
  `enodia_sentinel/schemas.py` and is pinned by
  `tests/test_schema_contracts.py::test_go_sidecar_v1_contract`.

## Last green verification

Run from the repository root:

```bash
python3 -m unittest discover -s tests -q
cd go-agent
GOCACHE=/tmp/enodia-go-cache go test ./...
GOCACHE=/tmp/enodia-go-cache go test -race ./...
GOCACHE=/tmp/enodia-go-cache go vet ./...
go mod verify
cd ..
make parity-go
make check-go-service
git diff --check
```

Results at original save time (`a25c73a`):

- Python: 377 tests passed.
- Go unit and race suites: all packages passed through `a25c73a`.
- `go vet` and `go mod verify`: passed.
- Parity: 22 poll alerts, 5 exec alerts, 7 syscall alerts, 10 typed-host
  alerts, 22 catalog records, 3 mixed-stream alerts, and 2 correlations
  through `88d7a6e`; the later integrity-anchor change is enrichment-only.
- `systemd-analyze verify` and the three Go sidecar packaging/isolation tests:
  passed.
- Static builds: `linux/amd64`, `linux/arm64`, `linux/riscv64`, and
  `linux/s390x` passed.
- Integration smoke tests passed for static binary identity, JSON emission,
  READY/STOPPING notification, heartbeat freshness/staleness, retained event
  equivalence, tail selection, corruption rejection, and evidence reading with
  malformed TOML. Snapshot smoke tests also passed for same-second batching,
  severity aggregation, private JSON/text pairs, truthful status counts,
  startup/quiet-period retention, process context, and Python dashboard-reader
  compatibility.

Results re-verified 2026-07-24 with the FIM/pkgdb/rootcheck engines and
go-sidecar dashboard consumer added on top:

- Python: 385 tests passed (`unittest discover`).
- Go unit and `-race` suites: all packages passed, including the new `fim`,
  `pkgdb`, and `rootcheck` packages.
- `go vet` and `go mod verify`: passed.
- Parity counts are unchanged from the original checkpoint (22/5/7/10/22/3/2),
  as expected: the new engines are integrity checks outside the poll/exec/
  syscall/host-rule parity fixture.
- `systemd-analyze verify` and the three Go sidecar packaging/isolation tests:
  passed.
- `git diff --check`: passed (no whitespace errors).
- Static cross-arch builds were not re-run this pass.

The Python suite prints three expected CLI error lines from negative tests and
one existing `ResourceWarning` for an implicitly cleaned-up HTTP 401 object;
the suite still exits successfully.

## Resume here

1. ~~Port the actual FIM/package-verification/rootcheck engines behind
   explicit, bounded acceptance gates~~ — done: `go-agent/internal/{fim,pkgdb,
   rootcheck}`, gated on `--state-dir`, tested, and wired into `agent.Sweep`.
2. ~~Add a read-only management API consumer for isolated Go snapshot/event
   state while keeping Python authoritative~~ — done: `/api/go-sidecar/*` in
   `enodia_sentinel/web.py` plus the dashboard Sidecar tab, schema-checked and
   never controlling the Go process.
3. Continue broader Phase 3 rule metadata/state parity before considering any
   default-service or package cutover. This is the remaining item from the
   original checkpoint.
4. Notification fan-out for the Go sidecar (item noted above) is still
   unported; the FIM/pkgdb/rootcheck engines currently return alerts only
   through the shared JSONL/snapshot/incident pipeline, not push backends.
5. Consider whether `04-integrity.md` and `05-rootcheck.md` in
   `docs/behavior/` (currently "planned") should be written now that these
   engines exist on both sides, so future Go-side changes have the same
   reviewable behavioral contract as the poll detectors and event rules do.

Keep the validation sidecar opt-in, preserve its separate state tree, and keep
new attack/event primitives disconnected from destructive response paths.