etminan
Hardening

The one host that actually holds the trust.

Etminan's own stated threat model is explicit about this: the monitored fleet can be fully compromised and the design still holds, but the verifier device itself is different — it's where the approved baseline, the operator's signing key, and the tamper-evident audit log all actually live. A compromised verifier can suppress alarms exactly as effectively as the self-checking model this project replaces. This page is a worked set of recipes for hardening that specific host — distinct from etminan.dev's own site-server hardening (a different machine, serving a marketing website, already documented in this repo's site-server/README.md).

Before the recipes

What's actually on this box, and why it changes the calculus.

baseline.db (every approved measurement, the pending-review queue, the hash-chained audit log), the operator's Ed25519 signing key, and this verifier's own mTLS identity all live here. None of that is protected by Etminan's own cryptography once someone has a real shell on this specific box — the whole design assumes operator DB and CLI access on the verifier is the trust boundary, not something it defends against from the inside. Hardening this host is the part of the system that's deliberately left to the operator, not built into the software.

One fact worth knowing before anything else: etminan-verifier never listens for inbound network connections at all — it only ever connects out to each agent (mTLS) and, if you use it, to fetch the plugin catalog. That means the firewall section near the bottom of this page isn't a compromise; a verifier host genuinely needs zero inbound ports open except whatever you use for administration (SSH, almost always).

01 — SSH: key-only, then hardware-key 2FA

Two layers, not one — a stolen key alone shouldn't be enough.

Baseline

Key-only authentication, no root login

The same baseline this repo's own site-server installer already applies to the (lower- stakes) marketing-website host — apply it here too, on a host that actually matters more:

/etc/ssh/sshd_config.d/10-hardening.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
X11Forwarding no
AllowTcpForwarding no

AllowTcpForwarding no matters more here than on a typical box: this host holds the operator key, so a tunneled connection through it is a more attractive pivot point than usual. Only relax it if you have a specific, understood reason to.

Second factor

Hardware-key-backed SSH keys (FIDO2/U2F) — no separate PAM module needed

OpenSSH (8.2+) can generate SSH keys that are themselves backed by a FIDO2/U2F hardware token (a YubiKey or similar) — the private key material never leaves the device, and every use requires a physical touch. This gets you real two-factor SSH access (something you have, cryptographically enforced) without installing or configuring a separate TOTP/PAM stack:

# on your own machine, with the hardware key plugged in:
$ ssh-keygen -t ed25519-sk -O resident -O verify-required -f ~/.ssh/etminan-verifier-key
# -O verify-required forces a PIN/touch on every single use, not just key generation

Add the resulting .pub file to the operator account's authorized_keys on the verifier host as usual. If a hardware key genuinely isn't available, a software TOTP second factor via libpam-google-authenticator is the fallback — weaker (a stolen TOTP seed is copyable; a hardware key generally isn't), but still real:

$ sudo apt install libpam-google-authenticator
$ google-authenticator
# then in /etc/pam.d/sshd, add: auth required pam_google_authenticator.so
# and in sshd_config: KbdInteractiveAuthentication yes, AuthenticationMethods publickey,keyboard-interactive

Test a second, already-open session before logging out after either change — a typo in AuthenticationMethods or a misconfigured PAM stack can lock out every future SSH session, and this box (unlike the site-server) usually has no separate out-of-band console readily at hand.

02 — A restricted shell for remote CLI access

SSH access to run etminan-verifier shouldn't hand out a real shell.

Every trust-changing etminan-verifier command is signed and audited — but only if it's actually reached through the CLI. A full interactive shell on this box can bypass all of that: edit baseline.db directly with sqlite3, read the operator signing key off disk, or tamper with state.json, none of which the signed/audited command surface would ever allow. If an operator account only ever needs to run etminan-verifier commands remotely, don't give it a general- purpose login shell at all.

The simplest version: an SSH Match block that forces every connection for that account straight into etminan-verifier, regardless of what the client actually requested:

/etc/ssh/sshd_config.d/20-operator-forcecommand.conf
Match User etminan-operator
    ForceCommand /usr/bin/etminan-verifier $SSH_ORIGINAL_COMMAND
    X11Forwarding no
    AllowTcpForwarding no
    PermitTTY no

$SSH_ORIGINAL_COMMAND is whatever the client asked for after ssh operator@host — e.g. ssh etminan-operator@verifier-host baseline review arrives here as baseline review, and ForceCommand runs exactly etminan-verifier baseline review, nothing else. An empty original command (a bare interactive ssh, no arguments) still only ever reaches etminan-verifier with no arguments — which, per its own usage banner, just prints help and exits, never a shell.

PermitTTY no closes the one gap ForceCommand alone doesn't: without it, a client can still request a pseudo-terminal and get an interactive-feeling session driving the forced command, which matters little for a one-shot CLI but costs nothing to close.

This recipe follows standard, documented sshd_config mechanics (Match/ForceCommand are long-established OpenSSH features) but hasn't been re-verified against a live login as part of writing this page — treat it as a correct starting recipe to test yourself before relying on it, not as something this session confirmed end-to-end on a real host.

03 — noexec, nosuid, nodev on /tmp

Nothing on this host should ever need to execute from a world-writable directory.

etminan-verifier and every plugin it runs already read their config from named, known paths — nothing in its own operation legitimately needs to execute a binary out of /tmp. Mounting it noexec,nosuid,nodev closes off a common local-privilege-escalation/persistence pattern (drop a payload somewhere world-writable, then execute it) for free, at the cost of nothing this project actually needs:

/etc/fstab
tmpfs   /tmp   tmpfs   defaults,noexec,nosuid,nodev,size=512M   0   0
$ sudo mount -o remount /tmp
# verify it actually took:
$ findmnt /tmp
TARGET SOURCE FSTYPE OPTIONS
/tmp tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=524288k

If anything on this host ever legitimately needs to execute a temp file (uncommon for this project's own operation), it'll fail loudly with Permission denied rather than silently — easy to notice and diagnose, not a subtle breakage.

04 — Sandbox the verifier's own systemd unit further

It already drops root and runs as its own user — it can be boxed in tighter still.

deploy/etminan-verifier.service already runs as a dedicated, unprivileged user with NoNewPrivileges=true and only the two capabilities it actually needs (CAP_SETUID/CAP_SETGID, to drop plugin children to their own sandboxed user — see the Plugin API). It doesn't yet use systemd's own filesystem/ namespace sandboxing directives, which cost nothing extra to add and meaningfully shrink what a compromised etminan-verifier process could reach even before an attacker gets anywhere near a real shell:

/etc/systemd/system/etminan-verifier.service.d/10-sandboxing.conf
[Service]
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/etminan-verifier
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
MemoryDenyWriteExecute=true
LockPersonality=true
$ sudo systemctl daemon-reload
$ sudo systemctl restart etminan-verifier.timer
# confirm the next run still succeeds:
$ sudo systemctl start etminan-verifier.service && journalctl -u etminan-verifier -n 20

ProtectSystem=strict makes the entire filesystem read-only to this unit except ReadWritePaths — the one directory it actually writes to (baseline.db, state.json). MemoryDenyWriteExecute blocks the classic write-then-execute exploitation pattern in-process, on top of the filesystem-level noexec /tmp above.

05 — An independent heartbeat, outside this host's own control

"The timer didn't fire" needs a witness that isn't this box itself.

etminan-verifier.timer firing hourly is only meaningful if something notices when it doesn't. A compromised or simply broken verifier can't be trusted to honestly report its own silence — the same reasoning this project already applies to a monitored host going quiet (silence is a finding) applies to the verifier itself. A dead-man's-switch-style heartbeat check, hosted somewhere this box doesn't control, closes that gap:

# add to cmd_run's own success path, or as a small wrapper around it — e.g. with Gatus
# or any dead-man's-switch service (healthchecks.io, Cronitor, etc.):
$ etminan-verifier run && curl -fsS --retry 3 https://hc-ping.com/<your-check-uuid>

Configure the external check with a grace period slightly longer than the hourly interval (e.g. alert if no ping arrives within 75 minutes) so a single slow cycle doesn't page anyone, but two consecutive missed cycles reliably does.

06 — An external anchor for the audit log's chain head

The hash chain has no anchor outside baseline.db today — give it one.

audit_log's hash chain proves nothing was quietly edited or reordered within the file as it exists now — but an attacker with raw write access to baseline.db itself could delete a row and recompute every hash after it into a fully self-consistent replacement chain that baseline verify-signatures would never flag. This is a stated, accepted limitation of the design (see Architecture & design), not an oversight — the fix is to periodically publish the chain's current head hash somewhere outside this file, so a rewrite has to explain away a value it can no longer reach back and change:

# cron, hourly, right after etminan-verifier.timer:
$ etminan-verifier baseline verify-signatures --print-chain-head \
| logger -t etminan-audit-chain -p local1.info
# then ship local1.* to a remote syslog aggregator/SIEM you already run,
# or append it to an S3 bucket with object-lock (write-once) enabled

--print-chain-head doesn't exist in etminan-verifier today — this section documents the intended shape of the fix (per the residual-risks note it closes), not a shipped command. Filed as its own roadmap item; the recipe above is what to wire up once it lands.

07 — Disk encryption at rest

Everything that matters on this host is two files.

/var/lib/etminan-verifier/baseline.db (the whole trust store) and the operator's Ed25519 signing key are the entire blast radius of this host being stolen, imaged, or having a decommissioned disk end up somewhere it shouldn't. Full-disk LUKS encryption, set up at OS install time, is the simplest complete answer — most Debian installers offer it directly during setup ("Guided — use entire disk with encrypted LVM"). If the host is already running without it, encrypting just the data volume after the fact is the realistic retrofit:

# provision a new, empty encrypted volume and migrate the data directory to it —
# in-place encryption of a live root filesystem is not something to attempt casually:
$ sudo cryptsetup luksFormat /dev/sdX1
$ sudo cryptsetup open /dev/sdX1 etminan-data
$ sudo mkfs.ext4 /dev/mapper/etminan-data
# mount at /var/lib/etminan-verifier, migrate the existing contents, add to /etc/crypttab + fstab

Encryption at rest defends against the disk itself being read outside a running, unlocked system — a decommissioned drive, a stolen machine, cold storage of a backup. It does not defend against a live, running host being compromised while mounted and unlocked; that's what every other section on this page is for.

08 — Default-deny firewall — zero inbound ports needed

The verifier never listens. The firewall should say so.

As noted at the top of this page, etminan-verifier only ever makes outbound connections (to each agent, and to the plugin catalog if you use it) — it has no listening socket of its own to protect. A default-deny inbound firewall that allows only SSH is the correct posture, not an inconvenience:

/etc/nftables.conf
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;
    ct state established,related accept
    iif lo accept
    tcp dport 22 ct state new limit rate 10/minute accept
  }
  chain forward { type filter hook forward priority 0; policy drop; }
  chain output { type filter hook output priority 0; policy accept; }
}
$ sudo systemctl enable --now nftables
$ sudo nft -f /etc/nftables.conf

output stays permissive here on purpose — this host needs to reach every enrolled agent's address, which isn't a fixed, enumerable list this firewall should try to hardcode. If your environment can enumerate every agent's address ahead of time, restricting outbound to exactly those destinations is a further, legitimate tightening beyond this baseline.