Add host correlation and assurance coverage

docs/COMMAND_REFERENCE.md

@@ -103,6 +103,8 @@ enodia-sentinel list-detectors
 ```
 
 Prints the poll detectors and whether each is enabled by config.
+The list includes rarity state such as `first_seen`; that detector keeps its
+local baseline under `log_dir/first-seen.json`.
 
 ### `rules`
 
@@ -282,6 +284,9 @@ sharing a PID or a common ancestor — the web server or SSH session they descen
 from), with a **time-window fallback** for alerts that carry no live PID (FIM
 drift, package tamper, hidden modules). Each captured snapshot records an
 `incident_id`; the index lives in `incidents.json` under `log_dir`.
+If a correlation rule matches the incident's raw alert set, the incident record
+also carries a `correlations` list and may escalate severity without hiding the
+underlying snapshots.
 
 - `list` — incidents newest-first: id, last seen, severity, alert count, and the
   signatures involved.
@@ -383,7 +388,7 @@ endpoint, for cron/monitoring.
 
 Stable JSON contracts are documented in [SCHEMAS.md](SCHEMAS.md). Current v1
 IDs cover alert objects/snapshots, incident records/views/exports, status,
-response plans, and response audit JSONL records.
+integrity/hash-chain state, response plans, and response audit JSONL records.
 
 Exit code:
 

docs/FLEET_DESIGN.md

@@ -0,0 +1,96 @@
+# Fleet Collector Design
+
+This note sketches the v1.2 fleet layer. It is intentionally optional: a local
+Sentinel agent must keep detecting, snapshotting, and serving its local console
+even when no collector exists or the collector is down.
+
+## Goals
+
+- Give a small fleet one view of host health, stale sensors, repeated
+  signatures, incidents, and integrity drift.
+- Preserve local autonomy. Detection, alert capture, response-plan generation,
+  and local evidence retention stay on each host.
+- Use append-only, auditable ingest. The collector stores what hosts report; it
+  does not execute response actions on agents.
+- Keep transport simple: HTTPS with pinned/self-signed deployments, or a
+  tailnet address where the operator already has one.
+
+## Non-Goals
+
+- No remote shell, remote command execution, or automatic containment in the
+  first fleet version.
+- No dependency on the collector for alerting or local dashboard availability.
+- No central trust reset. If a host is suspected compromised, the collector can
+  show evidence and staleness, but recovery remains an operator-reviewed local
+  workflow.
+
+## Agent-to-Collector Payloads
+
+Agents should push compact JSON records on a cadence and after important local
+events:
+
+| Payload | Source | Minimum contents |
+|---|---|---|
+| Heartbeat | `status --json` | host id, time, version, daemon state, heartbeat age, sensor states |
+| Alert summary | alert snapshot | snapshot name, incident id, severity, SIDs, signatures, counts |
+| Incident summary | `incidents.json` | id, host, timestamps, severity, SIDs, signatures, correlations |
+| Integrity summary | `/api/integrity` | watchdog, FIM/package anchors, reconciliation, hash-chain summary |
+| Hash-chain anchor | `hash-chain.jsonl` | latest `enodia.hash_chain.v1` record or last hash |
+
+Large evidence bundles stay local at first. The collector stores paths and
+digests so an operator can pull evidence deliberately.
+
+## Enrollment
+
+Each host gets:
+
+- a stable host id,
+- a per-host bearer token or mTLS client certificate,
+- an optional display name and tags,
+- a collector URL.
+
+Enrollment should write local config only after explicit operator approval. A
+lost token can be revoked collector-side without changing local detection.
+
+## Ingest API Shape
+
+Initial endpoints can stay narrow:
+
+- `POST /api/v1/ingest/heartbeat`
+- `POST /api/v1/ingest/alert`
+- `POST /api/v1/ingest/incident`
+- `POST /api/v1/ingest/integrity`
+- `POST /api/v1/ingest/hash-chain`
+
+Every request includes host id, timestamp, schema id, monotonic sequence number,
+and payload digest. The collector rejects unknown hosts, stale timestamps beyond
+policy, replayed sequence numbers, and schema ids it cannot parse.
+
+## Collector Storage
+
+A small SQLite database is enough for the first version:
+
+- `hosts`: identity, tags, token/cert fingerprint, last seen, enrollment state.
+- `heartbeats`: append-only health samples.
+- `alerts`: compact alert summaries keyed by host and snapshot name.
+- `incidents`: latest incident summary plus history rows for changes.
+- `integrity`: latest integrity state plus historical samples.
+- `hash_anchors`: host hash-chain tips over time.
+
+The collector dashboard should answer:
+
+- Which hosts are silent?
+- Which hosts have critical incidents?
+- Which hosts report repeated SIDs/signatures?
+- Which hosts have missing/stale integrity anchors?
+- Did a host hash-chain tip change as expected over time?
+
+## Security Notes
+
+- Treat collector data as sensitive forensic metadata; require TLS and auth.
+- Store per-host credentials hashed or certificate fingerprints only.
+- Do not let the collector mutate agent config in v1.2.
+- Keep host-side evidence and response actions local until a separate remote
+  write-path design is reviewed.
+- Consider mirroring `enodia.hash_chain.v1` records off-box before adding any
+  higher-risk fleet command surface.

docs/INDEX.md

@@ -26,6 +26,7 @@ same patch as any command, schema, or behavior documentation changes.
 | [SPECIFICATION.md](SPECIFICATION.md) | Product model, current scope, data model, and acceptance criteria. |
 | [ROADMAP.md](ROADMAP.md) | Release tracks and planned sequencing. |
 | [THREAT_MODEL.md](THREAT_MODEL.md) | Trust boundaries, attacker assumptions, and non-goals. |
+| [FLEET_DESIGN.md](FLEET_DESIGN.md) | Optional v1.2 collector design: enrollment, ingest, storage, and safety boundaries. |
 
 ## Versioning Rules
 

docs/ROADMAP.md

@@ -135,9 +135,11 @@ Purpose: reduce blind spots and connect related signals.
 - ✅ Add a second typed host-event rule family for `listen` events: SID `100068`
   flags interpreters opening listeners on unusual local ports, with local-port
   match conditions and a replayable fixture.
-- Continue generalizing the rule engine across more typed host events:
-  `bind`, `accept`, `file_write`, `chmod/chown`, `setuid/setgid`,
-  `capset`, and module-load events.
+- ✅ Continue generalizing the rule engine across more typed host events:
+  `file_write`, `chmod/chown`, and `setuid/setgid` now have built-in rules,
+  typed event parsing, fixtures, and rule-operation tests.
+- Continue extending typed host-event coverage to:
+  `bind`, `accept`, `capset`, and module-load events.
 - Keep rules declarative and Snort-like:
   `sid`, `msg`, `severity`, `classtype`, `event`, and match fields such as
   process name, parent process, argv regex, path prefix, peer IP/port, UID/GID,
@@ -146,7 +148,11 @@ Purpose: reduce blind spots and connect related signals.
   `tcp_connect`, `bind`, `accept`, module load, privilege transitions, and
   writes to watched persistence paths.
 - Build process lineage across polling and event sources.
-- Correlate multi-stage behavior:
+- ✅ Start correlating multi-stage behavior with a read-only incident
+  correlation engine: `exec_rule.web-rce` plus suspicious egress or an unusual
+  listener in a 10-minute incident window raises SID `100080`
+  `correlation.multi_stage_intrusion` while preserving the raw alerts.
+- Continue expanding multi-stage behavior:
   web service spawns shell, shell downloads payload, payload adds persistence,
   listener appears, and FIM changes.
 - Add configurable correlation windows and severity escalation rules, including:
@@ -162,9 +168,12 @@ Purpose: reduce blind spots and connect related signals.
   `curl|wget | sh`, Python/perl/bash reverse-shell argv variants,
   `systemctl enable` from unusual ancestry, `chmod +s` outside package
   transactions, and execution from writable directories.
-- Add first-seen and rarity tracking for local network behavior:
-  first public destination per process name, first listening port per binary,
-  interpreter connections to non-standard ports, and long-lived low-byte
+- ✅ Add first-seen and rarity tracking for local network behavior:
+  first public destination per process name and first listening port per
+  process name are locally baselined in `first-seen.json` and alert after the
+  initial silent pass.
+- Continue rarity coverage for:
+  interpreter connections to non-standard ports and long-lived low-byte
   connections when socket sampling can support it.
 - Keep polling as the oracle and fallback.
 
@@ -208,6 +217,9 @@ Exit criteria:
 Purpose: support several personal or small-business Linux hosts without turning
 the agent into a heavy enterprise platform.
 
+- ✅ Add a fleet collector design note covering optional collector boundaries,
+  per-host enrollment, signed JSON payloads, HTTPS/tailnet transport, and
+  standalone-agent failure semantics ([FLEET_DESIGN.md](FLEET_DESIGN.md)).
 - Add an optional collector service.
 - Agents push signed JSON events and heartbeats over HTTPS or a tailnet.
 - Collector stores host status, alerts, incidents, and package/integrity state.
@@ -230,7 +242,8 @@ Purpose: make tampering harder to hide from an attacker with high privileges.
 
 - External baseline anchor:
   mirror FIM and package DB fingerprints off-box with append-only semantics.
-- Hash-chain `events.log` and snapshots.
+- ✅ Hash-chain `events.log` and snapshots locally with append-only
+  `enodia.hash_chain.v1` JSONL records, exposed through the integrity report.
 - Optional snapshot signing with a host key.
 - Investigate Linux IMA/EVM and TPM-backed attestation for supported machines.
 - Replace bcc with a libbpf + CO-RE agent if the event layer becomes core to

docs/RULES.md

@@ -242,3 +242,78 @@ Expected false positives:
 - Administrative troubleshooting that temporarily listens on a high port.
 
 Drill or fixture: Use `rules test` with a `listen` event whose `comm` is an interpreter and whose `local_port` is not a common service port.
+
+## SID 100069: Interpreter wrote to a persistence path
+
+- Event: `file_write`
+- Signature: `host_rule.persistence-write`
+- Classtype: `persistence-write`
+- Severity: `HIGH`
+- Origin: `builtin`
+
+Match fields:
+- `events`: `file_write`
+- `comm`: `ash`, `bash`, `curl`, `dash`, `fetch`, `ksh`, `lua`, `nc`, `ncat`, `netcat`, `node`, `nodejs`, `perl`, `php`, `python`, `python2`, `python3`, `ruby`, `sh`, `socat`, `wget`, `zsh`
+- `path_prefixes`: `/etc/cron.d/`, `/etc/crontab`, `/etc/systemd/system/`, `/etc/ld.so.preload`, `/root/.ssh/authorized_keys`, `/etc/sudoers`, `/etc/sudoers.d/`
+
+Expected false positives:
+- Installer, configuration-management, or recovery scripts that legitimately write persistence paths from an interpreter.
+- Administrative SSH-key or service-unit maintenance performed by a reviewed script.
+
+Drill or fixture: Use `rules test` with `tests/fixtures/sids/100069-interpreter-persistence-write.json`.
+
+## SID 100070: Permissions or ownership changed on a persistence path
+
+- Event: `chmod,chown`
+- Signature: `host_rule.persistence-permission-change`
+- Classtype: `persistence-permission-change`
+- Severity: `HIGH`
+- Origin: `builtin`
+
+Match fields:
+- `events`: `chmod`, `chown`
+- `path_prefixes`: `/etc/cron.d/`, `/etc/crontab`, `/etc/systemd/system/`, `/etc/ld.so.preload`, `/root/.ssh/authorized_keys`, `/etc/sudoers`, `/etc/sudoers.d/`
+
+Expected false positives:
+- Package post-install scripts or configuration-management agents adjusting mode/owner on system service, sudoers, cron, or SSH key files.
+- Manual recovery work that repairs permissions after a known-good change.
+
+Drill or fixture: Use `rules test` with `tests/fixtures/sids/100070-persistence-permission-change.json`.
+
+## SID 100071: Interpreter requested a root UID transition
+
+- Event: `setuid`
+- Signature: `host_rule.privilege-transition`
+- Classtype: `privilege-transition`
+- Severity: `HIGH`
+- Origin: `builtin`
+
+Match fields:
+- `events`: `setuid`
+- `comm`: `ash`, `bash`, `curl`, `dash`, `fetch`, `ksh`, `lua`, `nc`, `ncat`, `netcat`, `node`, `nodejs`, `perl`, `php`, `python`, `python2`, `python3`, `ruby`, `sh`, `socat`, `wget`, `zsh`
+- `target_uids`: `0`
+
+Expected false positives:
+- Reviewed administrative wrappers that intentionally run interpreters across a root UID transition.
+- Privileged installer or service-management scripts during maintenance windows.
+
+Drill or fixture: Use `rules test` with `tests/fixtures/sids/100071-interpreter-setuid-root.json`.
+
+## SID 100072: Interpreter requested a root GID transition
+
+- Event: `setgid`
+- Signature: `host_rule.privilege-transition`
+- Classtype: `privilege-transition`
+- Severity: `HIGH`
+- Origin: `builtin`
+
+Match fields:
+- `events`: `setgid`
+- `comm`: `ash`, `bash`, `curl`, `dash`, `fetch`, `ksh`, `lua`, `nc`, `ncat`, `netcat`, `node`, `nodejs`, `perl`, `php`, `python`, `python2`, `python3`, `ruby`, `sh`, `socat`, `wget`, `zsh`
+- `target_gids`: `0`
+
+Expected false positives:
+- Reviewed administrative wrappers that intentionally run interpreters across a root GID transition.
+- Privileged installer or service-management scripts during maintenance windows.
+
+Drill or fixture: Use `rules test` with `tests/fixtures/sids/100072-interpreter-setgid-root.json`.

docs/RUNBOOKS.md

@@ -44,6 +44,29 @@ Then pick the runbook below that matches the signature.
 
 ---
 
+## Correlation Alerts
+
+**Triggers:** `correlation.multi_stage_intrusion` (sid 100080).
+
+Correlation alerts are not separate raw telemetry. They mean one incident now
+contains multiple lower-level signals that form a stronger story, such as a
+web/database service spawning an interpreter and the same incident showing
+suspicious egress or an unusual listener inside the correlation window.
+
+- Treat the correlated incident severity as the investigation priority.
+- Keep reviewing the underlying raw alerts; they remain in the incident
+  timeline and snapshots.
+- Export the incident before containment:
+
+  ```bash
+  enodia-sentinel incident export <incident-id> > /tmp/<incident-id>.json
+  ```
+
+- If the correlation involves a web/database parent, preserve service logs,
+  access logs, and application deployment state before restarting the service.
+
+---
+
 ## Runbook 1 — Reverse shell / suspicious egress
 
 **Triggers:** `reverse_shell` (sid 100010), `egress` (sid 100016),
@@ -359,9 +382,13 @@ suspicious case.
 
 - Keep the evidence bundle; do not delete snapshots while an investigation is open
   (retention is bounded by `max_snapshots` / `max_snapshot_age_days`).
+- Preserve `hash-chain.jsonl` with the snapshots. The chain records the local
+  digest sequence for snapshot logs, snapshot JSON, and `events.log`; a missing
+  or broken chain is not proof of compromise by itself, but it is a reason to
+  distrust local evidence and compare with off-box copies.
 - Re-baseline (`baseline`, `fim-update`) only once the host is confirmed clean —
   baselining a compromised host blesses the compromise.
 - Re-run `enodia-sentinel posture check` to close the hygiene gaps that enabled
   the intrusion.
-- The incident-grouping commands (`incident show/export`) on the
-  [roadmap](ROADMAP.md) will eventually automate the evidence-bundle steps above.
+- Use `enodia-sentinel incident show/export` and the saved response plan as the
+  durable handoff record for later review.

docs/SCHEMAS.md

@@ -66,6 +66,11 @@ Required fields:
 | `lineage` | array | Process-lineage IDs used for grouping. |
 | `snapshots` | array | Member snapshot filenames. |
 | `alert_count` | integer | Total alerts grouped into the incident. |
+| `correlations` | array | Higher-confidence correlation records derived from the incident's raw alerts. Empty when no correlation rule matched. |
+
+Each correlation record is additive evidence on the incident and keeps the raw
+alerts visible. Current fields are `sid`, `signature`, `classtype`, `severity`,
+`summary`, `window_seconds`, and `matched_signatures`.
 
 `incident show --json` returns `enodia.incident.view.v1` with `schema`,
 `incident`, and `timeline`. The dashboard incident API returns the same schema
@@ -108,8 +113,43 @@ Required fields:
 | `watchdog` | object | Daemon running state, heartbeat age, stale flag, max age, and verdict message. |
 | `anchors` | object | FIM baseline, package DB anchor, pacman keyring, and SigLevel summary. |
 | `sentinel_footprint` | object | Configured Sentinel self-integrity paths and present/missing counts. |
+| `hash_chain` | object | Local append-only hash-chain summary for snapshots and event logs: path, exists flag, record count, and last chain hash. |
 | `read_only` | boolean | Always true for this dashboard API. |
 
+## `enodia.hash_chain.v1`
+
+JSONL record appended to `hash-chain.jsonl` when Sentinel captures forensic
+artifacts. This is a local tamper-evidence chain; future fleet/external anchors
+can mirror the same records off-box.
+
+Required fields:
+
+| Field | Type | Meaning |
+|---|---|---|
+| `schema` | string | `enodia.hash_chain.v1`. |
+| `time` | number | Unix timestamp when the chain record was appended. |
+| `kind` | string | Artifact kind, such as `snapshot-log`, `snapshot-json`, or `events-log`. |
+| `path` | string | Local artifact path that was hashed. |
+| `size` | integer | Artifact size in bytes, or `0` if unavailable. |
+| `sha256` | string | SHA-256 of the artifact bytes, empty if unavailable. |
+| `prev_hash` | string | Previous chain hash, empty for the first usable record. |
+| `chain_hash` | string | SHA-256 over this record and `prev_hash`. |
+
+## `enodia.first_seen.v1`
+
+Local detector state stored in `first-seen.json`. The first detector pass builds
+a baseline silently; later passes alert on new public destinations per process
+name and new listener ports per process name.
+
+Required fields:
+
+| Field | Type | Meaning |
+|---|---|---|
+| `schema` | string | `enodia.first_seen.v1`. |
+| `initialized` | boolean | Whether the initial silent baseline pass has completed. |
+| `public_destinations` | object | Process name to sorted `host:port` destinations. |
+| `listener_ports` | object | Process name to sorted listener port strings. |
+
 ## `enodia.response.plan.v1`
 
 Dry-run response plan from `respond plan <incident-id>`.