# Etminan — TPM/IMA host-integrity attestation ## Sign and record every operator change (2026-07-21) — closes a real accountability gap, self-contained User requirement: "any change or approval must be documented so it can't be manipulated later." A direct read of every state-mutating `etminan-verifier` command found this wasn't true: `baseline exclude create` claimed a signing key but never actually signed anything (`exclusion_rules` had no signature column at all — `state.rs`'s own doc comment already admitted this); `baseline exclude revoke` took no `--key`/`--reason` at all, not even a claimed identity; `baseline reject` and `enroll` likewise had no operator key. `baseline approve` and `assign-profile` were genuinely signed, but nothing in this store was durable/append-only — a signed row proves its own content wasn't altered, but nothing proved the row (or the whole table) hadn't been deleted outright. **First attempt, reverted**: wiring every action into the existing `faaleoleo-audit` shellout (already used by `alarm.rs` for `run`/`reject` findings). Rejected: an external binary that may not be installed on a non-FDS host (exactly what Phase 6 is generalizing away from) is not a sound foundation for a guarantee this project is supposed to provide unconditionally, and it would have meant this project's core accountability story quietly depends on infrastructure outside its own crates — contrary to CLAUDE.md's independence requirement. **Actual fix — self-contained, in `verifier/src/audit_log.rs`**: a hash-chained, append-only `audit_log` table in `baseline.db`. Each row's `entry_hash` covers a canonical serialization of its own fields plus the *previous* row's hash (canonical JSON: sorted keys, no whitespace, minimal escaping — `verifier/src/canonical.rs`), so deleting, editing, or reordering any row breaks every hash after it, detectable via `baseline verify-signatures` (broadened in scope, not a new command — now also checks exclusion-rule signatures and the full `audit_log` chain, reporting pre-migration unsigned rows as a distinct third bucket rather than a failure). No new dependency: every crate this needs (`sha2`, `serde_json`, `ed25519-dalek`, `rusqlite`) was already in `verifier/Cargo.toml`. No new signing key either — a dedicated audit-log keypair (one automatic signature per row, independent of any operator action) was considered and rejected as disproportionate: it would add a new keygen step and a new "what if that key is missing" failure mode to defend against a threat this project's own existing code already puts out of scope (see the Threat model section's "verifier device" caveat, updated alongside this). Instead, every operator-attended row's `detail` embeds the *already-real* operator signature computed for that action — the hash chain's job is proving nothing was later removed or reordered, not re-proving authenticity a signature elsewhere in the same row already establishes. Six call sites in `verifier/src/main.rs` now sign (where they didn't) and append to `audit_log`: `baseline approve`, `baseline reject` (now requires `--key --reason`, previously took neither), `baseline exclude create` (now actually signs, previously only claimed to), `baseline exclude revoke` (now requires `--key --reason`, previously took neither), `assign-profile`, and `enroll` (now requires `--key --reason`, previously pure TOFU with no operator accountability record). `baseline.rs`'s `revoke_exclusion` also picked up a real bug fix along the way: it's now a single guarded `UPDATE ... WHERE revoked_at IS NULL` instead of a read-then-write, so a second `revoke` on an already-revoked rule errors instead of silently overwriting the first revocation's signature while leaving its timestamp unchanged — covered by a concurrency-style regression test, not just fixed by inspection. `cmd_run`'s findings also get a **best-effort** `audit_log` entry (`actor="system"`) alongside the unchanged `alarm::fire` call — best effort deliberately, since `run` is the unattended scheduled job and must never stop reporting a real problem because this secondary local-logging write failed. **Breaking CLI changes, pre-1.0** (see CLAUDE.md's existing tolerance for this): `enroll`, `baseline reject`, and `baseline exclude revoke` now require `--key`/`--reason` where they previously took neither. Corrects two stale claims elsewhere in this doc that predate this fix (see the Baseline workflow and Recommended-approach sections, both updated in place rather than left contradictory): `baseline approve` never actually called `faaleoleo-audit` despite Phase 2's write-up claiming it did, and a routine exclusion *match* during ongoing drift never got a `faaleoleo-audit` entry either (it only ever wrote to `excluded_log`) — decided out of scope here too, since a routine match isn't an operator change or approval, only rule creation/revocation is. **Not build-verified locally** — same recurring `tss-esapi-sys` native-Linux-only constraint as everything else in this doc (this session ran on macOS). Verified via `rustfmt --check` locally (syntax-level only); real verification is this project's usual path, Forgejo CI (`run-tests.yml`). New unit tests added: `audit_log.rs`'s chain/tamper/concurrent-append tests (the last mirrors `alarm.rs`'s pre-existing "two independent writers" class of regression test, applied to this new table), `signing.rs`'s action-changes-the-payload test, `baseline.rs`'s double-revoke regression test. ## Event-triggered write detection (2026-07-21) — closes the write-without-read gap's latency, not its existence User-identified gap: the existing mitigation for IMA's write-without-read blind spot (see "Confirmed gap" below) is a full read-sweep immediately before each hourly quote cycle — real, but it bounds detection to the polling interval. auditd-based approaches (which Tripwire/AIDE can layer on top via `perm=wa`) react more granularly to a write. Plain auditd doesn't actually close the gap in a way that matters here, though: its log isn't TPM-sealed, so it inherits the exact "root can suppress its own alarm" weakness this project exists to avoid — more granularity from a forgeable source isn't a stronger trust chain, just a faster forgeable one. **Implemented instead**: `agent/src/watch.rs`, an in-process fanotify watcher that triggers an immediate, *targeted* `sweep::sweep()` of just the changed file the instant a write completes, rather than waiting for the next hourly cycle. This changes *when* `sweep()` runs, never *what* gets trusted — the verifier's only evidence is still IMA→TPM; the watcher is a trigger, not evidence, so it adds no new attack surface to the trust chain itself. Two ways to scope the fanotify marks were considered and discussed with the user before implementing (a real privilege-boundary decision, per this project's own standing practice): - **Filesystem-wide mark + downstream filter** — mirrors the existing IMA policy's "measure broadly, filter by path on the verifier" pattern exactly, but needs `FAN_MARK_FILESYSTEM`, which always requires `CAP_SYS_ADMIN` regardless of kernel version. - **Targeted, unprivileged per-directory marks** (chosen) — needs no capability beyond the `CAP_DAC_READ_SEARCH` already granted for IMA log access. The target kernel (Debian trixie 6.12.95, confirmed in Phase 0) is well past Linux 5.13, where unprivileged fanotify notification-class events (`FAN_CLOSE_WRITE`, `FAN_MOVED_TO`, `FAN_CREATE` — everything this needs) stopped requiring `CAP_SYS_ADMIN`. Chosen for a reason beyond "smaller diff": it's a genuinely stronger contrast against Tripwire/AIDE-plus-auditd, which needs its own privileged daemon (`CAP_AUDIT_CONTROL`/`CAP_AUDIT_WRITE`) to get comparable write granularity. Etminan gets equal-or-better latency with **no new daemon and no new broad capability** — `deploy/etminan-agent.service`'s `AmbientCapabilities=` line is unchanged. Also keeps faith with this project's own "narrowest capability" precedent (see that same file's `CAP_DAC_READ_SEARCH` rationale comment). Implementation notes: - A watched **file** is marked via its **parent directory**, not its own inode — a fanotify mark follows the inode, not the path, so marking a file directly would go silently stale the moment that file is replaced via an atomic rename-over (a common pattern for config files and binaries during a package upgrade). The per-cycle sweep would still eventually catch a change like that; the point of this module is not needing to wait for it. - Directory marks use `FAN_EVENT_ON_CHILD` (report events on contents, not just the directory inode) and `FAN_ONDIR` + `FAN_CREATE` (report new subdirectories) — a new subdirectory gets an immediate mark of its own plus an immediate sweep, closing the classic recursive-watch gap for trees that grow after the agent started. - `watched_paths` is refreshed from `profiles::resolve_watched_paths()` every 60s inside the watcher loop, not on every single fanotify event — re-resolving reads `profiles.conf`/`assigned_profile` from disk, itself a real file read, and doing that on every event (which can fire far more often than hourly) would add self-inflicted IMA log volume beyond what already exists. - Fully best-effort at every layer: an old kernel, fanotify disabled, or one path's mark failing (e.g. `max_user_marks` exceeded) all degrade gracefully to per-cycle-sweep-only detection for the affected scope, with a logged warning. This feature is never required for the agent to start or keep serving quotes. - Known, accepted limitation: a path *removed* from a profile keeps its mark until the agent restarts — harmless (an extra sweep of a no-longer-watched file), not fixed in this pass. **Not yet build-verified locally** (same recurring `tss-esapi-sys` native-Linux-only constraint as everything else in this doc — this session ran on macOS) — the `nix` crate's fanotify API calls in `watch.rs` were written against its documented source (`Fanotify::init`/`mark`/ `read_events`, `FanotifyEvent`'s fd-ownership-on-Drop behavior) rather than compiled here. Per this project's established practice (see mTLS Progress below), verification should go through the project's real CI (`run-tests.yml`: fmt/clippy/build/test) rather than a local/manual build. Still open, beyond a green CI run: whether the CI runner's container environment permits the `fanotify_init` syscall for an unprivileged caller (Docker's default seccomp profile is a plausible blocker independent of kernel version) — worth an explicit check, not an assumption — and the live latency/fd-leak/dynamic-remarking behavior described in this project's plan file, which needs a real host with files actually being written to it, not just a clean compile. **Correction (2026-07-21) — the `nix`-based implementation above never actually worked, and has been replaced.** A code audit correctly flagged that `fanotify_init` above requests no `FAN_REPORT_*` flag at all. Checked against the real `fanotify_init(2)` man page (not memory, this file already had one unverified-API miss too many): since Linux 5.13, a fanotify group created *without* `CAP_SYS_ADMIN` **must** set `FAN_REPORT_FID` — plain fd-based reporting (what the original implementation used) requires `CAP_SYS_ADMIN` specifically, because handing an unprivileged process an open fd to an arbitrary marked file bypasses normal `open()` permission checks. The deployed unit only grants `CAP_DAC_READ_SEARCH`, so `fanotify_init` was failing in every real deployment — degrading gracefully to sweep-only, so nothing crashed, but the whole feature was silently dead. Worse than a one-flag fix: the `nix` crate's fanotify wrapper (confirmed against its source, not assumed) has **no support at all** for parsing FID-based events — only the plain fd-in-event format. `watch.rs` now hand-rolls `fanotify_init`/`fanotify_mark`/event-buffer parsing/ `open_by_handle_at` directly via raw `libc` calls (all confirmed present at the pinned `libc` version by name before writing any code), and the `nix` dependency is dropped from `agent/Cargo.toml` entirely — it was only ever added for this. The no-new-capability design goal still holds: `open_by_handle_at` (used to resolve a FID event's file handle back into a path) needs exactly `CAP_DAC_READ_SEARCH`, already granted. One more real nuance this surfaced: with `FAN_REPORT_FID` alone, a content event (`FAN_CLOSE_WRITE`) identifies the changed object directly, but a directory-entry event (`FAN_CREATE`, `FAN_MOVED_TO`) only identifies the *parent directory* — resolving the specific child needs additional info-record shapes (`FAN_REPORT_TARGET_FID`/`FAN_REPORT_DFID_NAME`) this implementation deliberately doesn't use. Since `sweep::sweep()` already recurses through directories safely, a parent-directory-only event sweeps the whole directory instead — correct either way, no name-string parsing needed, smaller implementation. The event-buffer parser itself is pure (operates on `&[u8]`, no syscalls) specifically so it's unit-testable against hand-built byte buffers without a kernel — see `watch.rs`'s test module. ## Centralized profile registry Progress (2026-07-21) — the deferred follow-on, now built Phase 6 (below) explicitly deferred a centralized `host_id -> profile` registry as "a separate follow-on decision, not an assumed extension of this work." That follow-on decision is what this is: with mTLS confirmed working end-to-end via real CI (see mTLS Progress below), the user asked to resume it, having already chosen "verifier pushes it to the agent" as the design when Phase 6 itself was scoped. - New `Request::SetProfile`/`Response::SetProfileAck` protocol messages (`common/src/protocol.rs`) — pushes a profile *selection* only, never the profile's contents, which stay in the agent's own local `profiles.conf`. Same connection-isolation invariant as `PackageLookup`: its own connection, never touches sweep/the TPM Context/`last_offset`. - New precedence tier in `agent/src/profiles.rs`: `ETMINAN_WATCHED_PATHS` > **a centrally-pushed profile name** (new) > `ETMINAN_PROFILE` > empty. The pushed tier wins over the local env var — otherwise centralizing selection would have no effect unless someone still hand-edited the local file to clear it. - **Watched paths are now re-resolved on every quote request, not once at agent startup** — the one real behavior change needed for a push to take effect without an agent restart. Confirmed safe: `watched_paths` only ever feeds `sweep::sweep()`, never PCR-replay/log-offset state, which is keyed purely on raw log byte offsets. A resolution error (missing file, unknown/dangling profile name) becomes a `Response::Error` for that cycle rather than crashing the agent. - New `etminan-verifier assign-profile --host --profile --key --reason ` — same `--key`/`--reason` bar as `cmd_baseline_approve` (changing what a host measures is the same class of trust-scope decision), signs a canonical payload the way `approve` actually does (not `baseline exclude create`'s weaker, identity-only pattern — confirmed by reading the code that `exclude create` never actually signs anything). The signature is for human-accountability/audit purposes on the verifier's own side; the agent doesn't independently verify it, since mTLS already gates *which verifier* may push at all. - Deliberately **no automatic re-push** from the scheduled `check`/`run` cycle — an early draft of this design added one and a review caught that it had no gate, meaning a new TLS handshake and an agent-side disk write every hourly cycle forever, even when nothing changed. Retry is a manual, safely-idempotent re-run of `assign-profile` if a push is suspected not to have landed. **Still true, unchanged**: this is per-host push-based assignment, not a new profile-*content* distribution channel — `profiles.conf` itself is still local to each agent. ## mTLS Progress (2026-07-21) — landed, transport now mandatory-mTLS Triggered by starting the centralized host→profile registry (below): a verifier-to-agent config push has no self-verification the way a quote response does, so the previously-tolerable plaintext transport became a real gap. Implemented pinned-fingerprint mTLS for the whole agent<-> verifier channel — see "mTLS (agent<->verifier transport security)" under Recommended approach for the full design, and `common/src/tls.rs` for the code. New `keygen-tls` subcommand on both binaries; `cmd_enroll` now pins a TLS fingerprint alongside the AK fingerprint in the same ceremony. **Since verified end-to-end via real Forgejo CI** (`run-tests.yml`: fmt/clippy/build/test all green for the whole workspace; the `.deb` built, GPG-signed, and published) — not build-verifiable in this dev environment itself (see the pre-existing `tss-esapi-sys` native-Linux-only constraint noted throughout this doc), so verification moved to the project's actual CI rather than a local/manual build. That verification is what unblocked the centralized profile registry follow-on (see above). ## Phase 6 Progress (2026-07-21) — watched-path profile registry started, out of phase order The user chose to start Phase 6 ("generalize beyond FDS") now, ahead of Phase 5 (optional FDS cutover), which is deferred to later — a sequencing choice, not a dependency the phase list itself changed. Investigation found the only real FDS-specific coupling remaining in the codebase was `agent/src/main.rs`'s `default_watched_paths()` — a hardcoded `fds-*` path list used whenever `ETMINAN_WATCHED_PATHS` was unset. Fixed: - **New `agent/src/profiles.rs`**: a named-profile registry (`/etc/etminan-agent/profiles.conf`, `[name]` sections, one path per line) so many hosts can share one reusable named path set instead of each host's `agent.env` inlining a full list. Hand-rolled, not a TOML/YAML crate — `agent` is the one crate in the workspace with no `serde` dependency, and this codebase already hand-rolls comparable small formats elsewhere (`ETMINAN_WATCHED_PATHS`'s comma-split, `dpkg -S`/`rpm -qf` output parsing, `/sys` file parsing) rather than reaching for a crate. - **Resolution precedence**: `ETMINAN_WATCHED_PATHS` (direct override) > `ETMINAN_PROFILE=` (looked up in the profiles file, failing loudly at startup if the file is missing or the name is unknown — a real misconfiguration) > empty, with a loud startup warning. **No hardcoded fallback of any kind** — that silent fallback to FDS-specific paths on a non-FDS host was exactly the bug this phase targets. - **Missing-path safety net**: every resolved path gets an existence check at startup, warning individually on any that don't exist — closes a gap where a misconfigured/placeholder profile (e.g. copying `profiles.conf.example`'s `generic-host` section verbatim) would otherwise resolve "successfully" and silently measure nothing. - Existing FDS deployments lose nothing: `deploy/profiles.conf.example` ships an `fds-host` profile byte-identical to the old hardcoded list — they now opt in explicitly via `ETMINAN_PROFILE=fds-host`, consistent with this project's "operator-approved only" philosophy applied to config too. **Boundary decision, stated explicitly**: this is **per-host env-var profile selection** — each host's own `agent.env` names which profile it uses, reusing Phase 4's existing per-host `EnvironmentFile=` delivery mechanism. This is **not** a centralized `host_id -> profile` registry (e.g. the verifier or a fleet config-management tool pushing assignments) — agents don't share a central config source today, and building one would be a materially bigger architectural change than this phase's own framing calls for ("mostly a config/UX surface, not a rearchitecture"). If fleet-wide centralized assignment is ever wanted, treat it as a separate follow-on decision, not an assumed extension of this work. **Update, 2026-07-21: that follow-on decision was made and built** — see "Centralized profile registry Progress" at the top of this doc. The mechanism described there still selects only, per host; it doesn't change anything about how profile *contents* are defined or distributed, which is what this boundary decision was actually protecting. **Not done** (as of this phase's own work, before the follow-on above): verifier-side changes (none needed for Phase 6 itself — the verifier has no FDS-specific coupling; "baseline scope" is already fully operator-driven via `exclusion_rules`, with no hardcoded defaults there either); build verification (blocked in this dev environment by the pre-existing `tss-esapi-sys` native-Linux-only constraint, same as every other session this project has hit that wall — reviewed by hand and via a Plan-agent design pass instead; since verified via real CI, see the mTLS section). ## Phase 4 Progress (2026-07-20) — packaging done; pilot not started Phase 4 is scoped much bigger than a code-writing sprint (AppArmor profile as a separate follow-on project, GPG-signed CI releases, and — most importantly — piloting in parallel with FDS through at least one real patch cycle, a time-based operational milestone that cannot be compressed into one session). Built and validated the concrete, testable part: real `.deb` packaging, exercised end-to-end, which surfaced three genuine bugs that every prior manual (always-as-root) test had been masking. - **`agent/src/ima_policy.rs`**: moved the manually-tested IMA policy content into real code — `etminan-agent write-ima-policy` auto-detects the root filesystem's UUID (via `findmnt`, not part of the quote-taking hot path so no risk of the spurious-measurement interference that rule exists to prevent) and writes the policy. Idempotent: treats "write node gone" or "write rejected" as success if measurements are already flowing, rather than failing — necessary because a loaded policy's write node **disappears from securityfs entirely** once consumed this boot (not just permission-denied on retry, which is what earlier Phase 0 testing had only observed until this session). - **`.deb` packaging**: one source package, both binaries (per the plan's scaffolding decision) — `debian/control`, `rules`, `postinst` (creates `etminan-agent`/`etminan-verifier` system users), `prerm`. Ships an early-boot systemd unit (`etminan-agent-ima-policy.service`, `Before= etminan-agent.service`) so the zero-reboot install claim is now real packaged behavior, not a manual step. - **Real bug #1 (postinst)**: `adduser --system` doesn't create a same-named group unless told to — the postinst's `chown etminan-verifier: etminan-verifier` failed on first install. Fixed with `--group`. Caught immediately by actually running `dpkg -i`, not by reading the script. - **Real bug #2 (TCTI device default)**: `tss-esapi`'s `DeviceConfig::default()` points at `/dev/tpm0` — the *raw* TPM device, group-owned by `root`. The agent runs as an unprivileged user in the `tss` group specifically to use `/dev/tpmrm0` (the resource-managed device, group-owned by `tss`) instead. Every manual test this whole project always ran as root, which can open both devices — masking this until the packaged systemd unit ran as the real unprivileged user for the first time. Fixed by explicitly constructing the device TCTI config with `/dev/tpmrm0` instead of relying on the crate's default. - **Real bug #3 (IMA log permissions)**: `/sys/kernel/security/ima/ ascii_runtime_measurements_sha256` is `root:root 0440` by kernel design — no group membership can grant an unprivileged user read access to it. **Stopped and asked the user before fixing this one** — it's a genuine privilege-boundary decision, not a mechanical bug fix. Resolved with `AmbientCapabilities=CAP_DAC_READ_SEARCH` in the systemd unit: the narrowest capability that bypasses read-permission checks specifically, confirmed by the user as preferable to running the agent as root outright. - **End-to-end confirmed**: after all three fixes, the packaged agent running as the unprivileged `etminan-agent` user (verified via `ps`/ `/proc//status` — genuinely not root) successfully produced a valid quote with no permission errors. The one remaining failure (`PcrDigestMismatch`) traced directly to `violations: 3` and a 6976-entry log on a host that had been running continuously for 2h25m through this session — the same pre-existing, already-documented violations/large-log issue from earlier phases, confirmed unrelated to the packaging fixes. **Not done**: the AppArmor profile (explicitly a separate follow-on project per the plan, and premature before the final product name is settled — an AppArmor profile targets specific install paths that would need to match final naming); GPG-signed CI release workflow (needs a real Forgejo repo, which doesn't exist yet for the same naming reason); the actual parallel-pilot-through-a-real-patch-cycle (fundamentally requires elapsed real-world time, not achievable in a single working session no matter how long). ## Phase 3 Progress (2026-07-20) Alarm path + `faaleoleo-audit` integration built and validated end-to-end — including a genuinely unexpected, complete success. - **`alarm.rs`**: best-effort shellout to `faaleoleo-audit log` (identical argument shape to FDS's `record_findings_via_faaleoleo_audit`: one shared `--correlation-id` per run, `--severity critical`, `--action security_incident`), plus a minimal `sendmail`-based email path — neither depends on `fds-core`, matching the project's independence requirement. Both degrade gracefully (warn, don't fail) if the audit key or `sendmail` aren't available. - **`run` subcommand**: the scheduled entry point (mirrors `security_check.rs`'s aggregate pattern). Iterates every enrolled host (address now recorded at enroll time — Phase 1/2's `HostState` didn't carry it, since only interactive single-host use existed then), collects findings, fires the alarm once for the whole run, and exits non-zero if anything was found or a check couldn't run at all. - **`baseline reject` now fires the alarm immediately**, not just a DB update — an explicit operator rejection is itself alarm-worthy per docs/design.md's Alarm path section. - **Deploy artifacts**: `deploy/etminan-verifier.service` (`Type=oneshot`, runs as an unprivileged `etminan-verifier` user — unlike FDS's security-check, this needs no root) + `.timer` (hourly, matching the scalability section's target interval) + `verifier.env.example`. **End-to-end validation — with a real surprise**: expected the `faaleoleo-audit` shellout to hit its graceful-degradation path (skip with a warning) since the binary wasn't expected to be installed on the test host. It turned out **`faaleoleo-audit` is genuinely installed and running** on `demo-host` already (as part of its existing FDS-adjacent setup) — so both the `baseline reject` test and the `run`-with-a-simulated- finding test (`ETMINAN_TEST_SIMULATE_REPLAY=1`) produced **real, signed, hash-chained audit entries**, confirmed by reading them directly out of `/var/log/faaleoleo/audit.jsonl`: ``` {"action":"security_incident","detail":{"_correlation_id":"attest-1784572428", "finding":"operator-rejected measurement: path=...", ...}, "resource":"demo-host","severity":"critical","source":"etminan-verifier", ...} ``` Exact structure match to what FDS's own security-check produces — this is the real integration working, not just the code path exercised. Separately confirmed `run` exits non-zero on a finding (tested via the built-in `ETMINAN_TEST_SIMULATE_REPLAY` hook rather than corrupting real state) — that's what a systemd unit would see as a failed run, closing the last leg of the Phase 3 exit criteria (audit entry + non-zero exit within one polling cycle; email alarm untested for real since no `ETMINAN_NOTIFY_EMAIL`/ mail transport was configured on this host — expected to hit the same graceful-degradation path FDS's own `send_via_sendmail` does when unconfigured). **Not done this phase, implemented later**: the "pending item aging past its SLA should escalate" behavior from this doc's baseline workflow section was an open policy question as of Phase 3. It's since been resolved with a 24-hour default SLA (`baseline::stale_pending`, called from `cmd_run` and `baseline review`) — **operator-configurable since 2026-07-21** via `ETMINAN_PENDING_REVIEW_SLA_HOURS` (`main.rs`'s `pending_review_sla_hours()`; unset/invalid/non-positive values fall back to the 24h default with a loud warning, never silently produce a threshold that flags nothing or everything as stale). Actually installing/ enabling the systemd units on a real host (as opposed to confirming the underlying command's exit-code behavior) was also not exercised this session. ## Phase 2 Progress (2026-07-20) Baseline workflow + operator approval built and validated end-to-end against a real host (`demo-host`). Implementation: - **Store** (`verifier/src/baseline.rs`, SQLite via `rusqlite`): `approved_measurements` (host_id, path, sha256, keyed on host+path), `pending_review`, `rejected`, `exclusion_rules`, `baseline_batches` (records each signed batch's payload/signature for audit), and `excluded_log` (every exclusion match still recorded, per docs/design.md's "never silently dropped" requirement). - **Per-file drift now tracked, not just the aggregate PCR**: `verify::check` (Phase 1) only ever compared the aggregate PCR10 value. Extended it to also return every `(path, sha256)` measurement entry covered by the matched log prefix — i.e. everything confirmed to have happened at or before the signed quote. That's what gets diffed against the baseline. - **Operator signing** (`verifier/src/signing.rs`): Ed25519, mirroring `fds_core::integrity`'s `load_signing_key`/raw-hex-seed convention but implemented independently (no `fds-core` dependency, per CLAUDE.md). One signature per batch, never per file — canonical payload is the sorted `(host_id, path, sha256)` tuples, so the same batch always signs identically regardless of iteration order. - **CLI**: `keygen`, `baseline review [--host]`, `baseline approve --host|--matching --key --reason`, `baseline reject --host --path --hash`, `baseline exclude create|list|revoke`. `--reason` is required on approve — the justification text is folded into the signed payload (see "More operator context" below), not just stored alongside it. **End-to-end validation results:** - First enrollment on a long-running host produced 2265 initial pending measurements; `baseline approve --host demo-host --key --reason "initial baseline for demo-host"` bulk-approved all of them in one signed batch — the "initial baseline" flow from docs/design.md working as intended. - Subsequent `check` runs correctly distinguish unchanged / pending / excluded, and legitimate new activity (even the verifier's own file reads/writes) lands in `pending_review` rather than being silently absorbed or falsely alarmed. - **Induced a real tamper** (wrote a new file under `/etc/fds`, a watched path) and confirmed it surfaced in `pending_review` on the very next check — never silently trusted. Rejected it via `baseline reject`; confirmed it left `pending_review` and landed correctly in the `rejected` table with all fields intact. This is the Phase 2 exit criteria, demonstrated directly rather than just implemented. - **Exclusion rule lifecycle confirmed**: created a rule for a known-noisy path (the verifier's own state file), confirmed a subsequent change to that path was excluded (not queued for review, but still logged to `excluded_log`) while an unrelated new item still correctly required review, then revoked the rule and confirmed its status flips to `REVOKED` in `baseline exclude list`. - **Not separately re-tested this session**: the cross-host `--matching` batch-approval variant (`baseline approve --matching --hash --reason `) uses the identical `approve_batch` path already exercised by the per-host flow above, just with a different SQL filter — validated by code review, not a live multi-host run, given time spent this session. ## Phase 1 Progress (2026-07-20) Core scaffolding + agent/verifier MVP built and tested end-to-end against a real TPM on the dev VM. All four exit-criteria scenarios validated: - **Genuine host**: `check` verifies signature, nonce, and PCR10 replay — confirmed reliably across 30+ consecutive real requests. - **Tampered/mismatched log**: any PCR-digest mismatch is caught and reported as a finding (same mechanism that also caught benign races during debugging — see below). - **Replayed quote**: nonce mismatch correctly detected (tested via `ETMINAN_TEST_SIMULATE_REPLAY=1`, which checks a real captured response against a nonce that was never actually sent — exactly what a replayed capture looks like from the verifier's side). - **Unenrolled AK**: `check` against a never-enrolled host ID correctly refuses with a finding rather than silently proceeding. **Real bugs found and fixed along the way** (all confirmed against a real TPM, not assumed): - Marshaled the *input template* passed to `create_primary` instead of `key.out_public` (the TPM's actual generated key) — the modulus field on a fresh template is deliberately empty; that's the TPM's job to fill in. - tss-esapi's `RsaSignature` struct has no public accessor for the signature bytes (confirmed against its source) — `signature_marshaled` wire bytes are the full `TPMT_SIGNATURE` (sigAlg + hashAlg + size + raw signature), not a raw PKCS#1v1.5 signature; hand-parsed the fixed 6-byte header instead of fighting that API. - `.as_bytes()` (via `zerocopy::IntoBytes`) on TPM2B-style buffer types reinterprets raw in-memory struct layout (including padding to the buffer's max size) — the real accessor is `.value()`, returning just the logical content, correctly sized. - **The big one**: an exact full-chunk log replay occasionally didn't match the signed quote, even though two independent implementations (Python and Rust) agreed with each other on the replay result. Root cause: the log file keeps growing from background system activity in the (however small) window between the quote being taken and the log being read afterward — the same class of race the Phase 0 spike found with an external CLI subprocess call, just now from genuine environmental noise. Fixed with **prefix-matching**: since the log only ever grows and the PCR only ever extends in lockstep, a valid quote's digest must match *some prefix* of what's received: anything after that prefix simply hasn't been reflected in a quote yet and belongs to the next cycle. Offset continuity between requests now tracks the matched prefix length, not the full chunk length, so unconsumed trailing entries correctly reappear in the next request's delta rather than being silently dropped (which would otherwise cause `cumulative_pcr10` to drift from the true PCR). - One reproducible large-log case (3393 entries, ~797KB chunk) showed no matching prefix at all even after this fix — both Python and Rust agreed on the full replay, and neither found any valid split point. Did not reproduce in ~30 subsequent normal-scale checks; noted as a residual edge case to watch for in Phase 2/pilot rather than fully root-caused here. **Resolved, 2026-07-21 — `tss-esapi` removed from `verifier`'s shipped dependency graph.** The deviation above (`verifier/Cargo.toml` taking on `tss-esapi` as a `[dependencies]` entry purely for its `Attest`/ `Signature`/`Public` marshal/unmarshal types, sharing the agent's build-time `libtss2-dev`/native-Linux-only constraint despite never touching a TPM at runtime) is fixed. New `verifier/src/tpm_wire.rs` hand-rolls the two structures actually used — `TPMS_ATTEST` (a `TPM2_Quote`'s signed attestation: nonce + signed PCR digest) and `TPMT_PUBLIC` (the AK's RSA public key) — replacing `Public::unmarshall`/ `Attest::unmarshall`. `TPMT_SIGNATURE` was already hand-parsed directly in `verify.rs`'s `check()` before this (tss-esapi's `RsaSignature` has no public accessor for the raw bytes), so this closes the last gap. Every field's offset/size was cross-checked against `tss-esapi-sys`'s own generated FFI struct layouts and `tss-esapi`'s own source (confirming `Public::marshall`/`unmarshall` operate on raw `TPMT_PUBLIC` bytes with no outer `TPM2B_PUBLIC` size prefix) — not assumed from memory. Verified independently before landing: a standalone scratch crate hand-built a `TPMT_PUBLIC` around a freshly generated real RSA key and confirmed the new parser's output verifies a real RSASSA-SHA256 signature — proof the derived wire layout round-trips with real cryptography. Deliberately strict rather than general-purpose: since this project's AK is created exclusively via one fixed, never-changing template (RSA/RSASSA-SHA256/ signing-only), `parse_rsa_public` rejects anything not matching that exact shape rather than handling every TPM algorithm branch a general library would — a legitimate agent never produces anything else. `tss-esapi` moved to `[dev-dependencies]` rather than being dropped outright — `verify.rs`'s existing test fixtures (`fake_ak()`, `build_attest()`) already marshal via the *real* library, so every existing test in that file became a genuine differential check of `tpm_wire.rs` against the authoritative implementation, for free. `cargo build`/cross-compiles never touch dev-dependencies at all, so this fully achieves the goal while `cargo test` (already Linux-CI-only for this crate, unchanged) keeps a real oracle as an ongoing regression guard. **Confirmed working, not just theorized** — for the first time this project's history, `etminan-verifier` now builds on this macOS dev machine (`tss-esapi-sys`'s build script no longer runs for a plain `cargo build`/`cargo build --release`), and a real `cargo zigbuild --release --target x86_64-unknown-linux-musl -p etminan-verifier` produced a genuine statically-linked musl binary. Also surfaced (and fixed, unrelated to this change but only ever visible once `etminan-verifier` could compile locally at all): a real borrow-checker error in `cmd_enroll` (a moved `String` reused after being moved into a struct literal) and several pre-existing `cargo clippy -- -D warnings` failures (dead-code fields on DB-row-mirroring structs, `entry_hash`'s argument count) — both crates (`common`, `verifier`) are now clean under `cargo clippy -p -- -D warnings` locally. Still needed, not done here: the live-host checklist every TPM-touching change in this project carries — confirm a real TPM's actual quote response parses identically before/after this change against a captured real quote, not just the synthetic/differential fixtures above. **Also tracked, not yet done**: the workspace still targets Rust edition 2021 (`Cargo.toml`'s `[workspace.package]`) rather than 2024 — never a deliberate constraint, just never revisited. 2024 changes temporary-drop- order and closure-capture rules in ways worth re-running the full test suite against rather than treating as a drive-by bump; do as its own follow-up, not bundled into an unrelated change. ## Second-host testing (2026-07-20) — a real finding, plus infra flakiness Tested against a second real host (a disposable clone, `Etminan-Dev-2`) to satisfy the exit criteria's "≥2 real hosts" requirement. Successfully enrolled and checked both hosts alternately from the same verifier multiple times, confirming cross-host operation works — but this host also surfaced a genuine, previously-undiscovered design gap: **IMA violations cause a PCR extend that is invisible to simple ascii-log replay.** Confirmed directly and repeatedly: `/sys/kernel/security/ima/violations` at 0 → replay matches the signed quote cleanly every time; violations at 1 → replay permanently diverges from the signed quote until the next reboot, with **no matching prefix at any point** in the log, confirmed independently in pure Python with no Rust code involved. This is a structural property of IMA, not a bug in this project's code: a violation extends the PCR through a mechanism that doesn't produce a normal log line, so no replay of normal log lines can ever reconstruct it. **Root cause narrowed further during the zero-reboot test below**: on this specific test host, the violation reliably accumulates within the *first minute* of boot — fast enough to make it worth searching harder for a cause. Checked thoroughly: no explicit message in `dmesg`/`journalctl -k`, no `auditd` running to log an `INTEGRITY_*` record, and no special/ distinct line in the ascii log itself (every line is a normal `ima-sig` entry). The likely explanation: this project's MVP policy (`measure func=FILE_CHECK mask=MAY_READ fsuuid=`) measures *every* file read across the entire root filesystem — deliberately broad because IMA has no path filter (see the corrected-policy section above), but broad enough that it can race with completely ordinary concurrent file access during a busy boot sequence (a service reading a file while another service is mid-write to it, etc.). If that's right, **violations aren't a rare edge case at all — they're close to an expected occurrence** on any reasonably busy host with this broad a policy, not just something that happens under attack or on one unlucky VM. **Design implication for Phase 2+, revised**: the verifier (or the agent, reporting it in the response) must read and track the violations counter per host, and a rising count is its own distinct finding — but if violations really are this common with a system-wide `FILE_CHECK` policy, the practical fix likely isn't just "handle violations gracefully downstream," it's **narrowing what gets measured** (e.g., scoping `FILE_CHECK` to specific known-quiet paths via `obj_type=` LSM labels instead of a blanket `fsuuid=`, accepting the AppArmor/SELinux labeling cost flagged earlier as "a much bigger lift than a plain path rule" — which this finding suggests may be worth paying) rather than trying to make broad measurement and boot-time noise coexist. This needs real investigation in Phase 2, not a guess baked into Phase 1. Separately, this second-host session ran into unrelated infrastructure flakiness worth recording so it isn't mistaken for a code issue later: a MAC address conflict between two unrelated VMs on the shared bridged network caused intermittent unreachability of the first host, and the second host's VM rebooted at least once without an explicit stop/start request during testing (cause not investigated further, given time spent). Neither is related to the attestation code itself. ## Zero-reboot clean-install test (2026-07-20) — partially confirmed Ran the install sequence in one pass on a freshly rebooted host with no prior policy write: securityfs IMA policy write (postinst-equivalent) → start agent (systemd-unit-equivalent) → enroll → check, with no intermediate reboot at any point. - **The policy-write half of the zero-reboot claim is confirmed**: on a genuinely fresh boot, the first securityfs write succeeds immediately, live, exactly as Phase 0 found — no reboot needed for that step, in this test or any earlier one this session. - **The full install-to-verified-enrollment flow was blocked by the violations issue above**, which (per the narrowed root cause) reliably hit within the first minute on this specific host/policy combination — faster than the whole install-and-enroll sequence could complete. - Net assessment: zero-reboot install is confirmed for the mechanism this project's code controls (the policy write itself); whether a *complete* zero-reboot install reliably reaches a verified state depends on resolving the violations/policy-breadth question above first. Carry both into Phase 2 together — they're the same underlying issue. **Not yet done**: removing the `ETMINAN_TEST_SIMULATE_REPLAY` env var path once a permanent automated test exists for it (currently a manual/documented test aid, not a security concern — it only changes what the *verifier* checks against locally, it cannot make an agent produce a falsely-matching quote); root-causing what specifically triggers the recurring violation on the second test host. ## Phase 0 Results (2026-07-20) — go/no-go: GO, with corrections Run against a dedicated dev clone (`Etminan-Dev`, cloned from the FDS demo image, UTM/QEMU backend with a `swtpm`-backed vTPM enabled — the live FDS demo/deploy VM was never modified). Summary for a reader who wants the outcome without the narrative: 1. **TPM connectivity + signed quote: proven.** A throwaway `tss-esapi` binary read PCR 10, created an AK under the Owner hierarchy, and requested a `TPM2_Quote` — real RSA-SSA-SHA256 signature, correct nonce echoed in `extra_data`, correct PCR selection. Core cryptographic mechanism works end-to-end from Rust against a real (software) TPM. 2. **IMA → TPM PCR extend: proven, causally.** Executing `fds-portal` for the first time changed `PCR10` from `E54F3C59…` to `623F0EE7…` (`tpm2_pcrread` before/after), with a matching log entry appearing at the same time. This is a real hardware-level effect, not just a log file. 3. **The core anti-forgery check itself: proven, byte-for-byte.** Replayed the IMA log's SHA256 template hashes through the PCR-extend algorithm and the result exactly matched the `pcr_digest` inside a freshly requested, signed `TPM2_Quote` — confirmed twice. This is the single mechanism that makes a fabricated log paired with a real signature detectable; it's now empirically validated, not just designed. Caught one real gotcha along the way: the generic IMA log file uses legacy SHA1-sized hashes regardless of PCR bank — the verifier must read the per-algorithm log file (`ascii_runtime_measurements_sha256`) matching the quoted bank. 4. **AK needs no persistent storage — proven across a real reboot.** The EK (a `TPM2_CreatePrimary` key) came back byte-identical before/after a full VM restart; our actual AK-creation template (also `TPM2_CreatePrimary`, restricted signing key) produced byte-identical output across two separate calls in one boot. Together: the agent can re-derive its AK fresh every boot with no key persistence needed, as long as its creation template never changes. (Caught a related trap: `tpm2_createak`'s CLI helper uses `TPM2_Create`, not `TPM2_CreatePrimary` — genuinely non-deterministic, confirmed by a mismatched AK across reboot before this was isolated. Our design doesn't use that path.) 5. **Verifier needs no TPM of its own: proven.** Used `tpm2_checkquote` (which has no TCTI/TPM-device flags at all) to verify a real quote using only the exported AK public key — exit 0, success. A negative control with a wrong nonce correctly failed ("Verify signature failed!", exit 1). Confirms the core architectural split: hosts need a TPM, the verifier device never touches one. 6. **`obj_path=` does not exist in real IMA policy — plan corrected.** The original design's path-filtered `measure func=... obj_path=/usr/local/ bin/fds-proxy` rules are invalid syntax; confirmed via direct kernel rejection (`EINVAL`) with the identical rule minus `obj_path=` accepted immediately. See the corrected policy design inline above (measure broadly, filter by path on the verifier — the log retains full paths regardless of policy scope). 7. **IMA policy is single-write-per-boot** (successful writes only; failed writes don't consume the slot) — confirmed by direct testing, not assumed. Production policy must be assembled as one complete write early at boot. 8. **Write-without-read coverage gap: confirmed, and its fix validated.** A new file under `/etc/fds` produced zero log entries until read; overwriting an already-measured file and never reading it again left the log showing the stale, pre-overwrite digest. Tested the fix directly: an active read sweep (`find ... -exec cat {} +`) immediately before each attestation cycle reliably picks up every pending change, including a second distinct overwrite — closing the gap at exactly the polling interval's granularity, no need to keep auditd running in parallel. 9. **No EK certificate chain on this TPM** (vendor reports generic "IBM SW TPM" — `swtpm`'s software implementation) — expected for a software TPM, confirms the plan's enrollment section is right that this VM class needs TOFU-with-a-witness bootstrapping, not certificate-chain validation. 10. **musl-vs-gnu build resolution: investigated and de-risked (2026-07-20), not yet implemented.** The org's `runs-on: docker` CI label is an ephemeral, root-by-default `node:20-bullseye` container with free `apt-get` access — already used for FDS's own musl release builds — so adding `libtss2-dev` there for a native-gnu `agent` build needs no new persistent infra. The three existing `Dockerfile.ci` files in the monorepo are confirmed vestigial (never wired into any workflow), so that was correctly ruled out as the path. Still needs an actual workflow file written and run once the project exists — carry into Phase 1 as the first build-system task. ## Further exploration (2026-07-20, same session) — additional validation Beyond the initial go/no-go, several more foundational claims were tested directly rather than left as architectural reasoning: - **Latency baseline measured**: ~225ms per host round-trip on this software TPM (`create_primary` + `quote` combined). Explicitly a lower bound, not representative of production — real hardware TPMs are commonly 5-10x slower for RSA operations. Still useful: confirms the operation isn't inherently a bottleneck even scaled up generously, informing the "Verifier scalability" section above. - **PCRs 0-7 confirmed meaningful, not placeholder zeros**: all eight UEFI boot/firmware PCRs hold distinct non-zero values on this VM (PCR3/ PCR6 coincidentally match — both get the standard `EV_SEPARATOR` event on most UEFI firmware, not a bug). Quoting them alongside PCR10 provides real additional signal, as the design intends. - **Multi-PCR quoting (PCR0-7 + PCR10 together) works correctly**: verified the combined `pcr_digest` is `SHA256` of all nine PCR values concatenated in ascending index order — independently recomputed from live `tpm2_pcrread` output and matched the quote's digest exactly. - **Full end-to-end tamper-detection simulation, reproduced 3x**: (1) establish a baseline hash for a watched file, (2) an "attacker" overwrites it without reading it back, (3) the agent's normal cycle (read-sweep, then quote) runs, (4) the freshly-measured hash is compared against the approved baseline → a clean `FINDING: changed since approved baseline` result every time. This is the complete pipeline working together, not just its individual mechanisms in isolation. - **Implementation-discipline lesson, not a design flaw**: while building the harness for the above, a subprocess call to the external `tpm2_pcrread` CLI tool *between* requesting a quote and reading the log turned out to be itself a measured event (under the broad `BPRM_CHECK` policy), occasionally injecting a spurious log entry that made a naive replay comparison mismatch — reproducible on the first call to a since-policy-reload-unmeasured binary, silently disappearing once that binary became an IMA cache hit. Root cause is specific to shelling out to a separate CLI process for a PCR read; the real Rust agent reads PCR0-7 through the same in-process `tss-esapi` session already open for the quote, so this exact interference cannot occur there. Worth carrying into Phase 1 as an explicit implementation rule: **the agent must never shell out to a separate TPM-touching CLI process between assembling a quote response and finalizing it** — everything PCR/quote-related stays in-process. - **Confirmed harness-only, not environment**: the earlier "IMA gap not reproducing" symptom mid-exploration traced to the IMA policy having been silently reset by an intervening reboot (expected one-shot-per-boot behavior, already documented above) and simply needing reapplication — not a new finding, but a live demonstration of exactly the "zero persistence across reboots" behavior the plan already accounts for. Everything above ran on `Etminan-Dev` (UTM name "FDS IDS", 192.168.10.84) — a dev copy of the FDS demo image, confirmed by the user to be a deliberate copy for this project, not the live `FDS.utm` instance (which was never touched). That VM was stopped/started three times and had `tpm2-tools`/`libtss2-dev`/`build-essential`/`cargo`/`rustc` installed and a live IMA policy written — each restart and install step was called out and separately confirmed with the user before running, consistent with this being shared/production-adjacent infrastructure rather than a fully disposable box. Worth renaming to `Etminan-Dev` in UTM (currently still "FDS IDS") for clarity going forward — flagged, not yet done. ## Context FDS currently detects host/package tampering via `fds-admin security-check` (`crates/fds-admin/src/cmd/verify_integrity.rs` + `check_host_changes.rs`), which re-hashes signed packages/images and watches FDS binaries/units/config via auditd, alarming from the same host it's checking. AIDE/Tripwire-style tools have the same weakness: the checksum database and the alarm both live on the box being watched, so a root-level compromise can suppress its own alarm before anyone sees it. The user wants a fundamentally stronger design, with **full root compromise** as the explicit threat model: a separate, independent device periodically (hourly) pulls integrity data from each host and does the comparison and alarming *itself*, so a compromised host can never silence its own alert. Because a userspace agent can be patched or lied to by root, checksums must come from something root cannot forge after the fact — that means Linux IMA measuring into a TPM 2.0 PCR (a one-way, kernel-level operation) plus remote TPM-quote attestation, the same pattern used by Keylime/RHEL attestation. "Self-learning" means drift is proposed, never auto-accepted — only an explicit, signed operator approval turns a change into the new baseline. This becomes a new standalone, fpm-tracked project, **not** an FDS crate, and it must develop and operate **entirely in parallel with FDS, with zero changes to FDS's existing code or `security-check` mechanism**. That's structurally natural here: the agent+verifier work at the OS/TPM layer, external to whatever they're protecting, so nothing about building or piloting this system touches `Faaleoleo-Distribution-Service` at all. The near-term goal is a **functional drop-in replacement** for the host-file- tamper half of FDS's `security-check` — proving it covers the same ground without requiring any FDS change to do so. Whether/when to actually retire the old auditd-based checks is a **separate, explicit, future decision**, never assumed or scheduled as part of this project (`verify_integrity`'s package/image signature checks are a distinct supply-chain concern anyway and may be kept regardless — open question either way). Longer-term, once proven against FDS, the stated goal is to generalize this into an **independent, FDS-agnostic platform** usable to monitor other software/hosts across the org — not an FDS-specific tool that happens to live in its own repo. The naming and architecture below are already chosen with that in mind (org-wide `Etminan` prefix, no `fds-core` dependency, generic host/measurement data model), but concretely parameterizing it for non-FDS targets is deliberately deferred, not designed now (see Phasing). Initial scope targets **Linux** (Linux IMA is the measurement mechanism); **BSD support is a stated future goal**, not current scope. BSD has no direct IMA equivalent, so the on-host measurement code must sit behind an abstraction rather than being hardcoded to Linux IMA, to avoid a redesign later (see Component architecture). TPM 2.0 (physical or vTPM) is confirmed available on target hosts. ## Recommended approach **Build a lean custom Rust agent + verifier using the `tss-esapi` crate — the same TSS 2.0 ESAPI bindings Keylime's own Rust agent is built on. Do not adopt Keylime wholesale.** Keylime is a full platform (its own DB, REST API, ZeroMQ revocation notifier, policy language) built around its own baseline-approval philosophy; retrofitting this project's "operator-approved only" philosophy and its own self-contained, signed audit trail on top of it is more work than building a minimal tool designed around those requirements from the start, and it's a much larger "new software" footprint. `tss-esapi` itself is a narrow, auditable TCG-spec binding — using it means trusting the TPM ecosystem's reference stack, not a third-party product's architecture. **Flag explicitly for prior approval before Phase 1 starts** (org policy — none of this is pre-cleared): adding `tss-esapi`/`tss-esapi-sys` + the `tpm2-tss` C library to the build toolchain and as a runtime dependency on hosts; enabling kernel IMA on hosts that don't already have it (a production security-control change, not just an app deploy); and the decision not to adopt Keylime. ### Component architecture Two independent trust domains — **the monitored host never judges itself**. **Host agent (`etminan-agent`)** — runs as a narrowly-privileged, non-root user (`etminan` + `tss` group for `/dev/tpmrm0`). On request, calls `tss-esapi`'s `Context::quote()` for PCR 10 (IMA) plus PCRs 0–7 (boot/firmware, cheap partial mitigation against below-IMA tampering), with a verifier-supplied nonce as `qualifyingData`, signed by the host's Attestation Key. Returns the quote + the IMA measurement log since the last fetched offset. **Makes no pass/fail decision, writes no audit entries, sends no alerts** — it's a relay, not a trust anchor. A compromised host can refuse to answer or kill the agent, but cannot fabricate a freshly-nonced, validly-signed clean quote, because the AK private key never leaves the TPM. Silence itself is a detectable, alarm-worthy state (see Alarm path). **No persistent AK key storage needed — proven in Phase 0, not assumed.** The AK is created via `TPM2_CreatePrimary` (a *primary* key, deterministic by TPM 2.0 spec design from the hierarchy's persistent seed + a fixed template — not `TPM2_Create`, which is what the generic `tpm2_createak` helper uses and which draws fresh randomness every time, confirmed non-deterministic in a side-by-side test). Directly verified: the EK (also `TPM2_CreatePrimary`-derived) came back byte-identical before and after a full VM reboot, and two separate `TPM2_CreatePrimary` calls with our actual restricted-signing-key template produced byte-identical keys within the same boot. Together this confirms the agent can simply **re-derive the same AK from the TPM on every boot** using a fixed template — no context-saving, no persistent key file, no loss-of-key recovery story needed. The one requirement this imposes: the agent's key creation code must use an unchanging, hardcoded template (same algorithm, attributes, key size) for the life of a host — changing the template would change the derived key and break the enrollment fingerprint match. **Verifier device (`etminan-verifier`)** — runs on a separate, independently-administered device; no TPM of its own required (quote verification is pure signature-verify math against the enrolled AK public key — **confirmed in Phase 0** with `tpm2_checkquote`, which has no TCTI/TPM-device flags at all: verified a real quote successfully using only the exported AK public key, exit 0; a negative control with a wrong nonce correctly failed with "Verify signature failed!", exit 1. Neither call touched `/dev/tpm*`). Single binary, subcommands `enroll` / `check` / `baseline review|approve|reject`, scheduled hourly (`etminan-verifier.timer`, mirroring `deploy/fds-security-check.timer`). Per host, each run: generate a fresh nonce → request quote → verify AK signature → confirm the embedded nonce matches (replay protection; **do not** rely on TPM clock fields, they aren't trustworthy across resets) → **replay the returned IMA log and recompute PCR 10, confirming it equals the digest inside the signed attest structure** (this is what ties the human-readable log to the cryptographic quote — without it, a host could pair a validly signed quote with a fabricated log) → diff against the approved baseline. **Empirically proven in Phase 0 (2026-07-20), not just designed**: replayed `ascii_runtime_measurements_sha256`'s template hashes through `PCR_new = SHA256(PCR_old || template_hash)` starting from 32 zero bytes, then hashed the result again (single-PCR selection), and it **exactly matched** the `pcr_digest` field inside a freshly requested, TPM-signed quote — full byte-for-byte match, confirmed twice. One critical detail this caught: the generic `ascii_runtime_measurements` file uses legacy SHA1-sized template hashes regardless of which PCR bank is in use — the verifier must read `ascii_runtime_measurements_sha256` specifically (or whichever per-algorithm file matches the PCR bank being quoted), or the replay silently computes the wrong value against the wrong bank. **Enrollment (the honest hard problem)**: prefer TCG Credential Activation where a manufacturer EK certificate chain exists. Many vTPMs won't have one — be explicit that vTPM attestation proves "not tamperable from inside the guest OS" (the stated threat model) but *not* "not software-emulated" or safe against a compromised hypervisor. Practical bootstrap: a deliberate, signed, human-in-the-loop enrollment at provisioning time — the operator records the new host's AK fingerprint into the verifier using their own signing key (same convention as `fds_core::integrity::load_signing_key`/ `FDS_SIGNING_KEY`). Trust-on-first-use, but the "first use" is a witnessed, signed operator action — same philosophy the baseline-approval flow uses, applied one layer earlier. **Auto-registration for new hosts (requirement confirmed with the user 2026-07-20, design only — not yet built)**: manually copying an AK fingerprint per host doesn't scale to fleet provisioning. Resolved as a **bootstrap-token model**, not open auto-trust — the human decision point moves earlier rather than disappearing, so this doesn't weaken the enrollment trust bootstrap called out below as the design's single weakest link: - Operator (or a provisioning pipeline acting on the operator's behalf) pre-generates a signed, single-use enrollment token *before* a host is provisioned: `etminan-verifier enroll token create --label --ttl 24h`. The token is a random secret plus a signature under the same operator key used for baseline approval — not derived from anything the host itself produces, so it can be minted with zero knowledge of the host's eventual AK. - The token gets baked into the host at provisioning time (e.g. cloud-init user-data, a provisioning image, or a secrets-manager pull) alongside wherever `etminan-agent` itself gets installed. - On first boot, the agent (or a thin bootstrap step ahead of it) presents the token plus its freshly-derived AK public key/fingerprint to the verifier's enrollment endpoint. The verifier auto-completes enrollment **only if**: the token signature checks out, the token is unexpired, and the token has not already been consumed (one token = one enrollment, enforced by marking it spent in the same transaction that records the new host). - A consumed/invalid/expired/replayed token is a rejected enrollment attempt, logged the same way an alarm-path finding is (this is itself a security-relevant event worth an audit trail, not a silent failure) — but does **not** escalate to the critical alarm path by default, since an expired token from a slow provisioning pipeline is routine, not an incident; a *repeated* invalid-token attempt against the same or a never-issued token is the pattern worth alarming on. - Newly auto-enrolled hosts still start with an **empty baseline** and still go through the existing first-baseline bulk-approve step — the token proves "an operator authorized a host to exist here," not "an operator has reviewed its current file state." Those stay two separate signed actions. - Tokens are listable/revocable (`etminan-verifier enroll token list|revoke`) so an operator can invalidate an unused token if a provisioning run is aborted or a token leaks before use. - This still doesn't solve the underlying EK-certificate-chain gap noted below — a token proves authorization, not hardware provenance. On hardware with a real manufacturer EK cert, TCG Credential Activation should still be preferred over token-based trust when available; the token model is specifically the vTPM/no-cert-chain fallback path, automated. **IMA policy — corrected by Phase 0 testing (2026-07-20) against a real kernel (Debian trixie, 6.12.95), superseding the original path-filtered design below**: stock upstream IMA policy has **no path-glob filter at all**. `obj_path=` was a design assumption that turned out to be invented — the kernel rejects it outright with `EINVAL`, confirmed live: an identical rule without `obj_path=` was accepted immediately. Real IMA policy conditions are limited to `func=`, `mask=`, `fsmagic=`, `fsuuid=`, `uid=`, `euid=`, `fowner=`, `fgroup=`, and LSM-label conditions (`obj_type=`, which would require tagging specific files with AppArmor/SELinux labels — a much bigger lift than a plain path rule). **Corrected design**: measure broadly and filter by path downstream, on the verifier, when parsing the log — each log line already carries the full path regardless of policy scope, confirmed live (`/usr/local/bin/fds-admin`, `/usr/local/bin/fds-portal` etc. all appear correctly in `ascii_runtime_measurements`). Policy actually verified working: ``` measure func=BPRM_CHECK mask=MAY_EXEC measure func=FILE_CHECK mask=MAY_READ fsuuid= ``` `fsuuid=` scopes `FILE_CHECK` to the root filesystem to avoid measuring every read system-wide; the verifier's baseline/diff logic (not the kernel policy) is what narrows this down to FDS binaries/`/etc/systemd/system`/ `/etc/fds` specifically. Confirmed live: **IMA policy accepts exactly one successful write per boot** (a second write attempt after a valid first write fails with the *same* `EINVAL`-shaped rejection, not a permission error) — a failed write does not consume the slot, only a successful one does. Production deployment must assemble the complete policy in one write at early boot (`/etc/ima/ima-policy` loaded via an early boot hook), not incrementally. **Explicit install-time requirement: zero reboots in the common case.** The Phase 0 spike needed three VM restarts, but none of that reflects the real install path — that was one one-time TPM-attachment step (irrelevant to a freshly provisioned host, where the TPM is present from the first boot by construction) plus two rounds of live debugging overhead from getting the policy syntax right. Confirmed directly: a securityfs policy write that is the *first* write of a boot succeeds immediately, live, no reboot needed. So `etminan-agent`'s `.deb` postinst can write the policy on install with zero reboot, and a shipped early-boot systemd unit repeats that write automatically on every future boot without operator involvement. The only case that genuinely needs a one-time reboot is a host that already had some other IMA policy loaded before our agent was installed (e.g. a distro default `ima_policy=tcb`) — same category of one-time bootstrap cost as installing any other kernel-adjacent security agent, not a repeated operational burden. **Phase 1 exit criteria should include an explicit "clean install, zero manual reboots" test on a freshly provisioned host**, to keep this a hard requirement rather than an assumption. **Confirmed gap, not just a risk to validate**: IMA measures purely on open-for-read; it has no write-triggered hook at all (unlike auditd's `perm=wa`, which fires on every write regardless of subsequent access). Directly tested: writing a brand-new file under `/etc/fds` produces **zero** log entries until it's read; overwriting an already-measured file and never reading it again leaves the log showing the **stale, pre-overwrite digest** — the tampering is genuinely invisible to IMA until something happens to read that file. When a read *does* follow a write, IMA correctly picks up the new content (confirmed: two distinct log entries for two distinct writes, each picked up on its own subsequent read) — so the gap is specifically "no proactive detection of a write," not "stale caching poisons future reads." **Recommended resolution — validated in Phase 0, not just proposed**: have the **agent actively read every in-scope file right before each attestation cycle** (a deliberate `open()`+read sweep over the FDS binaries, `/etc/systemd/system`, `/etc/fds`, immediately before requesting a quote). Tested directly: a file written-but-unread stayed invisible to IMA exactly as before, but a `find ... -exec cat {} +` sweep immediately picked it up; overwriting again without reading kept showing the stale first-sweep digest until the *next* sweep, which then correctly produced a distinct new log entry for the new content. This forces a fresh IMA measurement every polling interval regardless of whether anything else naturally reopened the file, closing the gap at exactly the granularity the whole design already targets (hourly) — without requiring auditd to keep running in parallel forever. Confirmed as the right default for Phase 1 rather than the auditd-parallel fallback. **Portability**: the agent↔verifier wire protocol (nonce request → quote + log-since-offset response) is already OS-agnostic. To avoid a redesign when BSD support is picked up later, the on-host measurement code should sit behind a `MeasurementSource` trait (in `common` or agent-local), with `LinuxImaSource` as the only implementation for now. BSD has no direct IMA equivalent — a future BSD backend is genuinely unresearched and stays explicitly out of scope until Linux is proven; the trait boundary exists only so that research doesn't require touching the protocol or verifier later. Likewise, IMA policy (the path list above) should be read from a per-host config/profile rather than hardcoded into the agent binary, so generalizing to non-FDS hosts later is a config change, not a rebuild. ### Baseline / self-learning workflow - **First baseline**: right after enrollment, an initial full quote+log pull presents every entry as an "initial candidate"; the operator reviews and bulk-approves in one sitting (`baseline review --host `) — still an explicit signed action, just covering a whole known-good image at once. - **Data model**: `approved_measurements(host_id, path, sha256, baseline_version, approved_by, approved_at)`. Each *review session* (not each file) is signed once as a batch (Ed25519, mirroring `fds_core::integrity`'s convention) — one signature per review, not one per file. - **Drift**: each hourly run diffs the fresh log against the current baseline; new/changed entries land in `pending_review`. This is a low-severity "N items awaiting review" notification, distinct from a critical alarm — mirrors FDS's own split between `SecurityCheckErrorEmail` (a check crashed) and `SecurityCheckAlertEmail` (a check found something). Never silently swallowed, never treated as an incident by itself. - **Approve** (`baseline approve --host --batch --key --reason `) signs and commits the batch, then records `action="baseline_approved"`, `severity="info"` in `audit_log` (distinct action/severity from an incident — this is an authorized change; see "Sign and record every operator change" below). `--reason` is required and folded into the signed payload (see "More operator context" below) — the justification is tamper-evident, not just a stored string. - **Reject** flags the entry as a confirmed incident, feeding the alarm path. - No silent timeout auto-accept or auto-reject. A pending item aging past a configurable SLA should escalate its notification severity rather than persist quietly. **Resolved as of Phase 3, made operator-configurable 2026-07-21**: `baseline::stale_pending`'s threshold defaults to 24 hours, overridable via `ETMINAN_PENDING_REVIEW_SLA_HOURS` (both `cmd_run` and `baseline review` read the same configured value, so the escalation finding and the review output's `[SLA EXCEEDED]` marker always agree). **Two more approval primitives, kept deliberately distinct** so fixing alert fatigue never quietly becomes "auto-accept everything": - **Cross-host batch approval** (retroactive, one-time): a fleet-wide patch or config rollout produces the *same* pending change across many hosts — reviewing host-by-host doesn't scale and pushes toward rubber-stamping. `baseline approve --matching --hash --reason ` groups every pending-review entry across the fleet that matches, shows the operator the full list of affected hosts before signing, and commits one signed batch covering all of them. This only clears the *currently* pending occurrence — it says nothing about the next time that path changes to a *different* hash, which still lands in `pending_review` normally. - **Standing exclusion rules** (proactive, forward-looking): for a path (or narrowly-scoped pattern) that's legitimately expected to change routinely — a rotating cache file, an embedded-timestamp config, anything that was in scope but never should have needed content-pinning — an operator can create a signed, audited suppression rule: `baseline exclude create --path --reason --key `. From then on, drift matching that rule is measured and logged (never silently dropped from the record — every match is still written to `excluded_log`, so there's still a forensic trail) but does **not** land in `pending_review` and does **not** need repeat approval. Creating or revoking the rule itself is what lands in `audit_log` (`baseline_exclusion_created`/`_revoked`) — see "Sign and record every operator change" below; a routine *match* against an existing rule is a system observation, not an operator action, so it isn't. Exclusion rules are themselves reviewable (`baseline exclude list`) and revocable (`baseline exclude revoke`), and scoping should default to exact paths — a broad glob is a bigger attack-surface decision and should be flagged more prominently when created (e.g. a confirmation showing how many currently-measured files the pattern would match). - Neither primitive touches the "operator-approved only" principle from requirement 5: both are explicit, signed, audited operator actions. The difference from plain approval is *scope* (many hosts at once / forward-looking) and *purpose* (clearing backlog / eliminating known recurring noise), not *who* decides. **More operator context (added post-Phase-2)**: a plain hash diff doesn't tell an operator *why* a file changed — IMA measurements carry no process, actor, or change-record information. Three additions close part of that gap: - **File metadata**: the agent attaches best-effort `stat()` data (uid/gid/ mode/mtime/size) to each measured path in the same quote round trip — purely informational, never part of the trust decision (`verify::check` never sees it). - **Package-manager correlation**: for pending (new/changed) paths, the verifier separately asks the agent whether an installed dpkg/rpm package owns the path, and which version. This is a *second*, on-demand connection, deliberately never pipelined with quote-taking — running `dpkg-query`/`rpm` is itself a subprocess exec, hence a new IMA-measured event, and must never risk delaying or corrupting the timing-sensitive quote path. Package checkers are pluggable (`agent/src/package_lookup.rs`'s `PackageChecker` trait) so another format (apk, pacman, ...) is a new impl plus one line, not a rework — the same plugin shape is the intended pattern for the future external change-management/ticket-system correlation mentioned below. - **Operator reason**: `baseline approve` requires `--reason`, folded into the signed batch payload (`signing::canonical_batch_payload`) so the justification is tamper-evident, not just a stored string. **Update, 2026-07-21: package-manager correlation deepened.** Plain ownership only proves *some* package claims a path — not that this exact version arrived via a real, timely, authenticated transaction, which is exactly the caveat `plain_language_verdict` already spelled out. Two additions, confirmed with the user, both computed agent-side in `package_lookup.rs` and reported back over the existing `PackageLookup`/`PackageInfo` round trip (`transaction_time`/ `trust_verified`/`trust_detail`, no new agent-verifier connection): - **Transaction timing**: `dpkg.log` (confirmed format: `YYYY-MM-DD HH:MM:SS : `) or, for rpm, `%{INSTALLTIME:date}` queried straight from the local rpmdb by package name — deliberately not `dnf history`'s output, which is front-end-specific (dnf/yum/dnf5 each render differently); the rpmdb value works regardless of which front-end actually did the install. - **Trust/provenance, honestly asymmetric by design**: researched before building, not assumed. RPM stores each installed package's original signature in its own database — `rpm -q --qf '%{SIGPGP:pgpsig}'` retroactively reports it with no original `.rpm` file needed (`rpmkeys(8)`'s `-K`/`--checksig` needs `PACKAGE_FILE` and operates on files, which is why the query-format path is used instead), cross- checked against `rpm -q gpg-pubkey`'s imported/trusted key IDs — a genuine cryptographic answer. dpkg/APT has no per-package signature at all in mainstream repos — only the repository's Release file is GPG-signed, checked once by APT at transaction time and not retained per-package afterward. The honest signal there is "did this go through a normal, authenticated APT transaction": a `dpkg.log` entry with no matching `/var/log/apt/history.log` block means it bypassed APT entirely (e.g. a raw `dpkg -i`); a matching block is then checked against `/var/log/apt/term.log` for the confirmed exact string `WARNING: The following packages cannot be authenticated!`. The verdict text (`plain_language_package_trust` in `main.rs`) never phrases dpkg's transaction-authentication signal as equivalent to rpm's real signature check — the confidence gap between the two ecosystems is surfaced to the operator, not smoothed over. Rejected alternative: independent agent-side re-verification against its own keyring, which would need original package artifacts generally not retained post-install plus agent-managed key infrastructure — infeasible for dpkg specifically, and a bigger lift than the value justified given RPM's own database already gives a real answer. - Not yet verified against a real host of each kind (Debian/Ubuntu, an RPM-based distro) — package-manager log formats can drift by distro/ version; flagged the same way every other log/API-parsing feature this project has shipped is. All three feed a single **plain-language verdict** in `baseline review` output — one sentence a generalist operator can act on (e.g. "Likely a legitimate install — matches package X v2.3.1" vs "⚠ An existing trusted file changed and is NOT owned by any installed package — this pattern is a common sign of tampering"), with the raw fields printed underneath for anyone who wants to double-check. The explicit goal: this tool's output must be actionable without escalating to a security architect. This bar should eventually be applied to `alarm::Finding` text and `verify::CheckOutcome` messages too — not done in this pass. **Update, 2026-07-21: built.** External change-management/ticket-system correlation (flag a pending change as outside any known approved change window) is implemented in `verifier/src/change_source.rs` — a source-agnostic `ChangeSource` trait, following the same in-process plugin pattern as `PackageChecker` above, with Request Tracker (RT) as the first example implementation (RT's REST2 API). Deliberately verifier-side only, unlike `PackageChecker`: a ticket system is externally reachable directly by the verifier, so this needed no agent changes and no `common/src/protocol.rs` messages. Every configured source is queried and every match is kept (not just the first) — an operator can have more than one system that could each confirm a change was planned, e.g. `ETMINAN_CHANGE_SOURCES=request-tracker,servicenow` once a second source exists — surfaced together in `baseline review`'s output as a new, independent line alongside the existing package- ownership verdict. Purely informational, per this section's own rule: never gates or auto-approves anything. See `docs/change-sources.md` for a full worked example of adding and configuring RT, and for what a second source implementation would need to provide. **Update, 2026-07-21: the external plugin-process model is built, superseding the in-process trait description above.** Every change source is now a separately-vetted external executable (any language), never in-process Rust — `verifier/src/change_source.rs` opens, verifies, and executes a plugin; it implements no ticket-system API itself. The explicit security requirement recorded below (written before this was built, kept verbatim as the record of what was required and why) is now enforced in code, checked fresh before every single execution, never cached from a prior run: - An explicit, operator-maintained allowlist (`change-source-plugins.conf`) — never a directory scan. A plugin not listed there by name is never executed, no matter what's on disk. - Ownership/permission check before exec: root-owned, not group/world- writable (mirroring how `sudo`/SSH refuse a world-writable config/key). - A pinned SHA-256 content hash per allowlisted plugin — a modified plugin (accidental or malicious) is refused, not silently run with different behavior. - The plugin file is opened once with `O_NOFOLLOW` and executed via `/proc/self/fd/` from that same already-verified descriptor — never re-resolving the path a second time, closing the TOCTOU window a naive check-then-exec-by-path would leave open to a symlink/rename swap. - A bounded timeout on plugin execution, so an unresponsive plugin can't hang a whole `run` cycle — this also closes what was an unbounded- blocking gap in the original in-process `ureq` HTTP call (no timeout configured), found in a later audit. Request Tracker (RT) is the reference implementation, shipped as `deploy/change-sources/request-tracker.sh` (shell + `curl` + `jq`) rather than Rust — see `docs/change-sources.md` for the full security model and setup, and its "Adding a second source" section for the plugin contract (host_id + anchor in via argv, a JSON array out via stdout). Nothing in `ChangeRecord`, the `change_correlations` storage, or the `baseline review` rendering needed to change for this — exactly as anticipated when this was originally deferred. **Original rationale, kept for the record**: an in-process Rust trait (mirroring `PackageChecker`) was built first because it shipped faster and proved the `ChangeRecord`/storage/rendering design end-to-end before committing to the external-process contract. That in-process version never had a network timeout configured on its HTTP call — a real gap, closed by this rewrite rather than patched separately, since moving the HTTP call into an external, timeout-wrapped process fixed it as a side effect of the architecture change rather than needing a second fix. **Explicit security requirement that drove this design (as originally written, before it was built — kept verbatim as the record of what was required and why)**: the verifier must never execute an arbitrary, self-registering external program as a change source. "Drop a script in a directory" must not itself be sufficient to get it run — that would let anything capable of writing to the plugin directory (or, on a shared host, a lower-privileged local account) get silently invoked with network access and its output trusted enough to print in `baseline review`. Whatever the eventual mechanism, it needs the same "operator-approved only" bar this project already applies everywhere else (the AK-fingerprint TOFU ceremony, the pinned TLS fingerprint, signed baseline approval): an explicit, operator-maintained allowlist of plugin paths (not a directory scan), plus an ownership/permission check before exec (root-owned, not group/world-writable — mirroring how `sudo` and SSH refuse to honor a world-writable config/key), and ideally a pinned content hash per allowlisted plugin so a compromised or tampered script can't silently start returning forged "a change was planned" confirmations. This is a real, deliberately higher trust boundary than a package-manager query (which only ever reads local package-manager state) — a change-source plugin can shape what an operator believes about whether a change was authorized. **Wire-protocol note**: adding package-lookup support required a breaking change to `common/src/protocol.rs` — the verifier now sends a tagged `Request` enum (`Quote` | `PackageLookup`) instead of a bare `QuoteRequest`. No version negotiation exists yet; agent and verifier must be redeployed together. Acceptable pre-1.0 (Phase 1/2), called out here so it isn't a surprise later. ### mTLS (agent<->verifier transport security) The transport was plain, unauthenticated TCP through Phase 1-5 — tolerable only because every message was either self-verifying (the TPM-signed quote) or purely informational (package-lookup answers). Adding a verifier-to-agent config-push feature (the profile registry, above) exposed a real gap: a push command has no equivalent self-verification, so an on-path attacker could silently reassign a host's config with nothing to detect it. Fixed by making the whole channel mTLS, mandatory, no plaintext fallback. **Trust model: pinned certificate fingerprints (SHA256), not a CA chain.** This is a closed two-party channel between one verifier and each of its enrolled agents — analogous to SSH host-key pinning, not a public CA-validated HTTPS service (unlike `fds-proxy`/`fds-portal` elsewhere in the org, which terminate real public TLS via Let's Encrypt because they *are* public-facing). It directly extends this project's own trust-on-first-use pattern: `cmd_enroll` already pins the TPM AK fingerprint, confirmed out-of-band by a human operator; the agent's TLS certificate fingerprint is now pinned in the exact same ceremony, printed and confirmed alongside it, and stored in `HostState. tls_cert_fingerprint_sha256_hex`. Every later connection (`check`/`run`) uses that pinned fingerprint; only `cmd_enroll`'s first-ever connection to a host is allowed to accept whatever certificate is presented. **Mutual, both directions**: the agent (TLS server) also authenticates the verifier (TLS client) — via `ETMINAN_VERIFIER_CERT_FINGERPRINT` in that agent's `agent.env`. Unset, the agent still requires mTLS (some client certificate, proof of key possession) but doesn't check *which* certificate, and prints a warning on every startup — never a silent gap, mirroring the "no watched paths configured" warning from the profile registry work. **Stack**: `rustls` 0.23 (pure Rust, no OpenSSL — matches the org's `fds-proxy`/`fds-portal` convention) with the `ring` crypto provider (avoids `aws-lc-rs`'s cc/cmake build-time dependency), `rcgen` for self-signed keypair generation. Synchronous (`rustls::StreamOwned`), no async runtime added. All of it lives in `common/src/tls.rs`, exposed only through opaque type aliases (`ClientTlsConfig`/`ServerTlsConfig`/ `TlsClientStream`/`TlsServerStream`/`CapturedFingerprint`) so neither `agent` nor `verifier` needs `rustls`/`rcgen` as a direct dependency — `agent` in particular stays as dependency-light as before this landed. **Key lifecycle**: `etminan-agent keygen-tls` / `etminan-verifier keygen-tls` (mirrors the existing `keygen` subcommand's shape) generate one self-signed keypair each, persisted under `ETMINAN_TLS_DIR` (`/var/lib/etminan-{agent,verifier}/tls` by default) and printed with their fingerprint for the operator to carry into the other side's config. No auto-generation on first use — an explicit, auditable step, consistent with how the operator Ed25519 signing key is already handled. **Key rotation (2026-07-21)**: the two directions of pinning are handled differently, on purpose. - **Agent's cert, pinned in the verifier's `state.json`** — the case with real per-host state and genuinely no update path before this: `etminan-agent keygen-tls` to regenerate, then `etminan-verifier rotate-tls --host --key --reason ` on the verifier. Structurally mirrors `cmd_enroll`'s first-ever connection (TOFU-captures whatever certificate is presented, since the new fingerprint isn't known yet) but is a continuation of an already-trusted host, not a fresh one: the connecting host's AK fingerprint must still match what's on record (same physical TPM — rotating the transport cert must never be usable to sneak a different, unreviewed host past AK validation) and the PCR10 replay continues from the stored cumulative value rather than restarting from zero. AK trust and the approved baseline are untouched — only `tls_cert_fingerprint_sha256_hex` changes, signed and recorded in `audit_log` (`action="tls_cert_rotated"`) same as every other trust-changing action. - **Verifier's cert, pinned statically per-agent in `agent.env`'s `ETMINAN_VERIFIER_CERT_FINGERPRINT`** — deliberately stays a manual procedure, not a new ceremony: `etminan-verifier keygen-tls`, then update `ETMINAN_VERIFIER_CERT_FINGERPRINT` in every agent's `agent.env` and restart each agent, in that order. Doing it any other order (or restarting an agent before its env is updated) locks that agent out until it's fixed — there is no dual-fingerprint transition window built (accepting either an old or new pinned value simultaneously), a deliberate scope decision (confirmed with the user): it's a bigger change to `common/src/tls.rs`'s verifier trust model for a case that's already just a config value an operator controls directly, and this project would rather keep that surface small than build a transition mechanism for something with a straightforward-if-manual safe sequence. - **AK rotation is out of scope entirely** — the AK is TPM-hardware-bound via a fixed, deterministic creation template (never `TPM2_Create`'s EK-child pattern — see the agent's own implementation notes), so a changed AK fingerprint means a different physical TPM. Genuine re-enrollment is the correct ceremony for that case, not something to rotate around. ### Verifier scalability — how many hosts per backend Not yet measured (no real verifier built yet); architectural reasoning only, to be validated against real hardware in the Phase 4 pilot. Per-host hourly cost is small and mostly I/O-bound, not CPU-bound: a network round-trip to the agent, the agent's own TPM quote operation (the one potentially slow step — real hardware TPMs are notoriously slow at RSA operations, often 300ms–1s per quote, vs. software/vTPM which is much faster, as observed in Phase 0), then on the verifier side just a signature verify (cheap) + an IMA log replay over the entries since last check (cheap) + a SQLite lookup/write. Polled concurrently rather than serially, a single verifier process on modest hardware could plausibly handle somewhere in the hundreds-to-low-thousands of hosts within an hourly window — an estimate, not a measured ceiling. **The real bottleneck is expected to be the operator-approval workflow, not throughput.** Every genuine baseline drift requires an explicit human-signed approval by design (a deliberate anti-normalization control, not an oversight) — so the practical ceiling on hosts-per-backend is really "how much baseline drift can an operator reasonably review per day," which scales with fleet patch/deploy cadence, not raw host count. **Segmenting verifiers by environment/trust tier is also a security argument, independent of throughput**: the verifier itself is a new high-value target (see Residual risks below) — concentrating an entire fleet under one verifier means one verifier compromise defeats everything it watches. Prefer multiple smaller verifiers (e.g. one for demo/staging, a separate one for production) over maximizing hosts-per-backend even if the hardware could technically take it. To actually validate in Phase 4 rather than assume: real TPM quote latency on physical hardware (not just `swtpm`), SQLite write throughput under concurrent multi-host polling, and realistic operator review cadence for a representative fleet size. ### Alarm path (runs only on the verifier device) **As actually implemented (2026-07-21): all four items below are built.** Multi-channel notification fan-out (item 3) superseded its original design — see "Notification channels: a shared external plugin framework" right after this section for what was actually built and why it replaced the `NotificationChannel` trait/enum this section originally sketched. Triggers: quote signature failure; PCR-recompute-from-log mismatch; unrecognized/replayed/expired nonce; **host unreachable within the polling window is itself a finding**, not a soft-skip (this is what defeats "just don't answer" as an evasion); an explicit `baseline reject`; a pending item aging past its SLA. On trigger, mirroring `security_check.rs`'s pattern in spirit (not its external-audit-tool integration, which this project deliberately doesn't have): 1. Local structured log on the verifier. 2. A durable, hash-chained `audit_log` entry (`action="security_finding"`, `actor="system"`) — self-contained, in `baseline.db`, no external key or binary; best-effort so a logging failure never blocks `run` from still reporting the finding (see "Sign and record every operator change" below for the full design). 3. Notification fan-out — multi-channel, not email-only. Email (`send_via_sendmail`, unchanged since Phase 3) plus PagerDuty, Slack, and a generic webhook, each independently enabled and independently filterable by severity (`ETMINAN_NOTIFY__SEVERITIES`) — see the next section for the actual (plugin-based, not trait/enum) architecture. 4. Non-zero exit from the verifier's systemd unit, so external unit monitoring (e.g. Gatus) is a second, independent failure-detection channel beyond the notification channels above. Explicit non-goal: no auto-remediation/auto-quarantine in this phase. ### Notification channels: a shared external plugin framework (2026-07-21) The original sketch above (a `NotificationChannel` trait/enum in `verifier/src/alarm.rs`, one enum arm per channel) was superseded before being built, for the same reason this project already ripped `ureq` (an in-process HTTP client) out of the verifier for change-source correlation in 0.5.0: an in-process Rust HTTP integration per channel means Etminan itself has to know about PagerDuty, Slack, and whatever a given customer happens to use — and, per the user directly, "we can't convince people to use another change management or ticket system," which applies equally to notification targets. What got built instead is the **`notify` category of a shared, formalized "Etminan Plugin API"** (see `docs/plugins.md`), the same external-plugin model change-source correlation already used, generalized: - `verifier/src/plugin_exec.rs` — the verify-then-exec security mechanics (open with `O_NOFOLLOW`, fstat root-owned/non-group-writable checks, hash the open fd against an allowlist's pinned SHA-256, exec via `/proc/self/fd/` under a timeout) extracted out of `change_source.rs` so both categories — and any future one — share one implementation instead of a second hand-copied TOCTOU-sensitive one per category. - `verifier/src/notify.rs` — the `notify`-specific parts: reads `ETMINAN_NOTIFY_CHANNELS`, loads `notify-plugins.conf`, and for each enabled+allowlisted+severity-matching channel, pipes the findings as a JSON array over the plugin's **stdin** (not argv, unlike change-source's `--host`/`--anchor` flags — finding text is free-form and argv-escaping it is exactly the injection surface to avoid). - Reference plugins: `deploy/notify-plugins/{pagerduty,slack,webhook}.sh`. PagerDuty's `dedup_key` is `attest-` as originally planned, so repeat findings for one host still correlate into a single incident. **A deliberate strengthening beyond the original sketch**: a broken or misconfigured plugin (either category) must never fail silently — it's the alerting mechanism itself, so a silent failure there is the worst possible failure mode. `etminan-verifier plugins verify` checks every currently-configured plugin on demand; `cmd_run` runs the same check automatically every cycle and turns a failure into an ordinary `warning`-severity finding (through the same print/email/notify/audit_log path as any other finding) rather than a stderr line that only shows up if someone happens to be watching. ### Project scaffolding Name: **`Etminan`** (org-wide prefix — the tool is host-agnostic long-term despite being FDS-motivated). Binaries: `etminan-agent`, `etminan-verifier`. Deviates from the single-crate `Bacula-Forgejo` template: use a **Cargo workspace** (`agent/`, `verifier/`, `common/`) so the verifier's build never pulls in `tss-esapi`/TPM bindings it doesn't need. ``` Etminan/ README.md, CHANGELOG, CLAUDE.md, gpg-faaleo-dev-team.asc, package.conf Cargo.toml # [workspace], members = ["agent","verifier","common"] agent/ Cargo.toml (deps: tss-esapi), src/main.rs, quote.rs, ima_log.rs, transport.rs verifier/ Cargo.toml (no tss-esapi), src/main.rs, enroll.rs, baseline.rs, alarm.rs, audit_shellout.rs common/ Cargo.toml, src/protocol.rs (nonce/quote wire types), src/ima_measurement.rs docs/, man/man1/{etminan-agent.1,etminan-verifier.1} debian/, rpm/ # one source package, both binaries; deployer enables # only the relevant systemd unit per host role .forgejo/workflows/{run-tests.yml,release-package.yml,release-deb.yml,release-rpm.yml} .forgejo/ISSUE_TEMPLATE/ ``` `package.conf`: ``` package_name=etminan install_prefix=/usr binary_dir=/usr/bin man_dir=/usr/share/man/man1 doc_dir=/usr/share/doc/etminan ``` Verify empirically (Phase 0/1) that a workspace root `Cargo.toml` without a `[package]` block still satisfies `.fpm/config.yaml`'s `required_files.rust` file-presence check for `fpm import` — don't assume. **musl cross-compile risk, real and concrete — resolution confirmed against the org's actual CI (2026-07-20)**: `tss-esapi-sys` links `tpm2-tss` (`libtss2-esys`/`-tctildr`/`-mu`, ≥2.3.3) via pkg-config; even its `bundled` feature produces a dynamically-linked result, not static-musl. The existing `cargo zigbuild --release --target x86_64-unknown-linux-musl` convention (used by FDS/fpm-source) does not work as-is for the `agent` crate. Investigated the org's real Forgejo CI setup rather than assuming a fix would fit: the `runs-on: docker` label used by every `release-deb.yml`-style job (including FDS's own musl zigbuild release) is an **ephemeral `node:20-bullseye` container, root-by-default, `apt-get` available with no `sudo` needed** — confirmed via `Docs-And-More/docs/forgejo/ 04-installing-and-enabling-the-forgejo-runner.md` and the workflow files themselves. That means the resolution needs **zero new persistent infrastructure**: add `apt-get install -y libtss2-dev pkg-config` to the existing ephemeral container job and target `x86_64-unknown-linux-gnu` (native, no zigbuild) for just the `agent` binary, while `verifier` keeps the existing musl path unchanged in the same workflow file. `debian/control` gets a `Depends:` on the dynamic `tpm2-tss` runtime libs for target hosts. Explicitly **not** the resolution: a custom `container:`-referenced `Dockerfile.ci` with `libtss2-dev` baked in. Three `Dockerfile.ci` files already exist in the monorepo (`Bacula-Forgejo`, `Bacula-GitHub-Cloud`, `Faaleoleo-Assistant`) but none of them are actually wired into any workflow — they're vestigial, not a proven pattern — so building on them would be genuinely new, unproven CI territory. The ephemeral-container `apt-get` approach reuses infrastructure this org already runs successfully today, which is the whole reason to prefer it. ### Phasing **Phase 0 — feasibility spike (throwaway code, explicit go/no-go gate) — COMPLETE, 2026-07-20, on `Etminan-Dev` (UTM/QEMU + swtpm, cloned from the FDS demo image; see Phase 0 Results below for the full report.** Minimal native-gnu `tss-esapi` binary: enumerate PCR banks, read PCR 10, request/verify a self-signed-AK quote — proven against UTM's swtpm-backed vTPM. IMA enabled, log entries confirmed landing and provably extending the real TPM PCR10 (before/after `tpm2_pcrread` diff around a fresh exec). `obj_path=` turned out not to exist in real IMA policy — corrected design above. Write-without-read coverage gap empirically confirmed (not just theorized) and a mitigation (active read sweep) identified. EK certificate chain confirmed absent on this swtpm instance, as expected for a software TPM — enrollment must be TOFU-with-a-witness for this VM class. **Not yet done, deferred to Phase 1**: musl/gnu build resolution against this org's actual Forgejo CI (only tested with the local Debian toolchain so far). **Phase 1 — core scaffolding + agent/verifier MVP** Create the project (layout above). Agent: quote-on-request over TLS, IMA log reader, active read-sweep of in-scope paths before each attestation cycle (closes the write-without-read gap from Phase 0). Verifier: manual-fingerprint enrollment CLI, scheduled poll, signature + PCR-recompute verification — no baseline diffing yet. Exit criteria: `etminan- verifier check --host ` reliably tells apart a genuine host, a tampered-log host, a replayed quote, and an unenrolled AK, against ≥2 real hosts; **and a clean install of `etminan-agent` on a freshly provisioned host (TPM present from first boot) requires zero manual reboots** — package postinst writes the IMA policy live, confirmed possible in Phase 0. **Phase 2 — baseline workflow + operator approval** sqlite baseline store, `pending_review`/`approved`/`rejected` states, review/approve/reject CLI, batch Ed25519 signing, initial-baseline bulk-approve. Exit criteria: an induced unauthorized change surfaces as `pending_review`, never silently trusted, only joins the baseline after explicit signed approval; rejection flows to the alarm path. **Phase 3 — alarm path + faaleoleo-audit integration** Full alarm path above: shellout, multi-channel notification (email + PagerDuty at minimum), timer, "host went dark" detection. Exit criteria: an induced verification failure produces a signed `faaleoleo-audit` entry, a notification on every configured channel appropriate to its severity, and a failed systemd unit within one polling cycle. **Phase 4 — hardening, packaging, IMA rollout, pilot (drop-in replacement, Linux, FDS paths) — zero changes to the FDS repo throughout** Follow-on `AppArmor-Etminan-Agent` project (separate repo, existing `AppArmor-FDS-*` convention). `.deb`/`.rpm` packaging, GPG-signed releases, IMA policy rollout targeting the FDS paths from Component architecture. Pilot on a small real host subset + one real verifier, **running fully in parallel with, and never modifying,** FDS's existing `verify_integrity`/ `check_host_changes`, through at least one full legitimate patch cycle, to confirm the approval workflow doesn't cause operator fatigue on routine changes. Exit criteria: sustained parallel operation with no false positives against real patch/deploy activity, and no FDS code touched to get there — this is the "proven drop-in replacement" milestone. **Phase 5 (future, separate decision — not committed) — optional FDS cutover** Only if/when explicitly decided later, tracked as its own change in the FDS repo, not scheduled as part of this project: remove `verify_integrity`/ `check_host_changes` from FDS's `security_check.rs` (keep `worker_liveness`/`garage_cleanup_backlog` — operational, out of scope), retire `deploy/fds-auditd.rules`, update `checks.rs` and docs. Open question for that future decision: `verify_integrity` also covers package/image Ed25519 supply-chain re-verification, distinct from IMA's runtime-tamper check — whether that half is retired or kept as defense-in-depth is independent of whether the host-file-tamper half is ever cut over. **Phase 6 — generalize beyond FDS (started 2026-07-21, out of phase order — see Phase 6 Progress at the top of this doc)** Turn per-host IMA policy scope into a config/profile the platform ships generically, so other projects' hosts can onboard without agent code changes. **Done**: a named watched-path profile registry (`agent/src/profiles.rs`, `/etc/etminan-agent/profiles.conf`) replacing the old hardcoded FDS default, selected per host via `ETMINAN_PROFILE` in that host's `agent.env`. **Also done, as a 2026-07-21 follow-on**: a centralized `host_id -> profile` *selection* registry — the verifier can push which profile a host uses (`assign-profile`) over the mTLS channel, without the profile's contents ever leaving that host's own `profiles.conf`. See "Centralized profile registry Progress" at the top of this doc for the full mechanism. **Phase 7 (future, out of scope for now) — BSD backend research** Investigate a BSD-side `MeasurementSource` implementation (no direct IMA equivalent exists — candidates to research, not commit to, include TrustedBSD/MAC-framework-based measurement or a custom kernel module). Explicitly deferred until Linux is proven in production. ### Residual risks — stated plainly, not oversold - **AK enrollment trust bootstrap is the weakest link** — without a full manufacturer EK-cert chain, initial trust is TOFU-with-a-human-witness, not provable back to hardware. Get the provisioning ceremony right; it's the foundation everything else stands on. - **vTPM guarantees are weaker than physical TPM** — defends against guest-OS-root compromise (the stated threat model), not a compromised hypervisor or infrastructure operator. Name that dependency explicitly per host class. - **Below-the-TPM-boundary compromise is out of scope** — firmware/UEFI implants or SPI-bus attacks against the measurement chain's own root of trust aren't caught. PCRs 0–7 give a cheap partial mitigation, not full boot attestation. - **Physical attacks are out of scope** (cold-boot, TPM desoldering, evil-maid) — this defends against remote/software compromise only. - **DoS is detectable, not preventable** — a privileged attacker can still firewall/kill the agent. Silence is correctly treated as alarm-worthy, but "no alarm" must never be read as "definitely fine," only "fine, or already flagged unreachable." - **The verifier device itself becomes a new high-value target** — it holds the baseline data and operator-approval trust. A compromised verifier can suppress alarms exactly as effectively as the AIDE-style model this project replaces; `audit_log`'s hash chain and each action's operator signature (see "Sign and record every operator change" below) are a detective control for a partially-compromised or tampered store, not a defense against a verifier that's fully compromised end-to-end — this project's stated stance is that operator DB+CLI access on the verifier is already the trust boundary. Concretely: the hash chain has no anchor external to `baseline.db` itself, so an attacker with raw write access to that file could delete a row and recompute every hash after it into a self-consistent replacement chain that `verify_chain` wouldn't flag — accepted as matching this stated boundary, not an oversight (confirmed 2026-07-21, see `audit_log.rs`'s own module doc comment). Recommend hardening/monitoring the verifier host at least as much as the fleet it watches, ideally with an independent third-party heartbeat check (e.g. Gatus) confirming its timer actually ran, and/or periodically publishing the chain's current head hash somewhere outside `baseline.db` (syslog, a remote log aggregator, immutable object storage) to give `verify_chain` a point it can't be silently rewritten past — both flagged as needed follow-ups, not designed in this plan. ## Critical files (reference, not modify — this is a new project) - `Faaleoleo-Distribution-Service/crates/fds-admin/src/cmd/security_check.rs` — pattern for the verifier's alarm path (audit shellout, email split, non-zero exit on findings vs. crashed checks). - `Faaleoleo-Distribution-Service/crates/fds-core/src/integrity.rs` — pattern for operator Ed25519 baseline-approval signing (`load_signing_key`/`FDS_SIGNING_KEY`). - `Faaleoleo-Distribution-Service/deploy/fds-auditd.rules` — coverage target for the new IMA policy. - `Faaleoleo-Distribution-Service/crates/fds-admin/src/cmd/check_host_changes.rs` — the check being replaced; useful reference for check-result plumbing. - `Bacula-Forgejo/package.conf`, `Bacula-Forgejo/Cargo.toml` — project scaffolding template (adapted to a workspace layout). - `Faaleoleo-Distribution-Service/.forgejo/workflows/release-deb.yml` — the musl/`cargo zigbuild` convention the `agent` crate must deviate from. ## Verification This plan's near-term output is Phase 0 (a feasibility spike), not shipped code — success is measured by concrete go/no-go findings, not a test suite: - A minimal `tss-esapi` binary builds natively (gnu target) and produces a verifiable TPM quote against `swtpm`, then a real target-class TPM/vTPM. - IMA enabled on one test host; log entries appear under `/sys/kernel/security/ima/ascii_runtime_measurements`; PCR 10 recomputed from the log matches a live quote's signed digest. - Documented answer (not assumption) on the `obj_path=`/auditd coverage- parity gap for `/etc/systemd/system` and `/etc/fds`. - Documented answer on what the org's actual vTPM implementation proves. - Findings reviewed with the user before Phase 1 scaffolding begins.