etminan
Notification channels

How an alarm actually reaches somebody.

etminan-verifier run always prints every finding and, if configured, emails it. PagerDuty, Slack, and a generic webhook ship as ready-to-use channels on top of that — each an external, allowlisted script under the same Etminan Plugin API as change-source correlation, never a built-in integration baked into the verifier binary. This page covers the full contract: what a channel receives, how to reshape its messages without touching its script, and exactly how little work adding a brand-new channel actually is.

01 — What this is

No in-process HTTP client, on purpose.

The verifier itself has zero built-in knowledge of PagerDuty's API, Slack's webhook format, or anyone else's paging system. Every channel beyond email is a separately vetted, root-owned, SHA-256-pinned external script — the exact same security model change-source (ticket-system) correlation uses, and for the same reason: a customer should never need a verifier rebuild to point an alert at their own on-call tool.

0

lines of Rust to add, remove, or reconfigure a notification channel. Every channel — shipped or custom — is enabled, disabled, filtered, and reformatted entirely through verifier.env and notify-plugins.conf. No code change, no rebuild, no redeploy of the verifier binary.

02 — Shipped channels

Three reference plugins, one generic escape hatch.

Install only the ones you're actually enabling — each is an independent, individually-allowlisted script.

deploy/notify-plugins/pagerduty.sh

PagerDuty

One Events API v2 event per finding. dedup_key = "attest-<host_id>" correlates repeat findings for the same host into one incident instead of paging fresh every cycle. kind rides along as a custom_details field PagerDuty's own routing rules can match on.

Needs: ETMINAN_PAGERDUTY_ROUTING_KEY

deploy/notify-plugins/slack.sh

Slack

One digest message for the whole batch — a noisy check cycle with many findings is still a single Slack message, one line per finding, not a flood.

Needs: ETMINAN_SLACK_WEBHOOK_URL

deploy/notify-plugins/webhook.sh

Generic webhook

Passes the findings array through unmodified as the POST body — any HTTPS endpoint that accepts JSON, and the template to copy when writing a fully custom channel.

Needs: ETMINAN_WEBHOOK_URL, optional ETMINAN_WEBHOOK_AUTH_HEADER

03 — Configuring a channel

Four steps, all outside the verifier binary.

etminan-verifier plugins install pagerduty (or slack/webhook) is designed to do steps 1 and 2 below at once — it verifies a catalog manifest's Ed25519 signature against a compiled-in trust anchor before trusting anything it lists, then downloads the script, hashes it, and writes the allowlist line itself. The sha256 pin still happens either way; catalog install just adds a second, cryptographic layer in front of it instead of it being the only thing standing between you and running someone else's script. Not yet usable in practice: the command and its verification are shipped in etminan-verifier, but nothing publishes a real, signed catalog manifest at its default URL yet — see the plugin directory for current status. Until that exists, this is the only real path:

Install the plugin script, root-owned

$ sudo install -D -m 0755 -o root -g root \
deploy/notify-plugins/pagerduty.sh \
/etc/etminan-verifier/notify-plugins/pagerduty.sh

Allowlist it with its exact content hash

$ sha256sum /etc/etminan-verifier/notify-plugins/pagerduty.sh

Add it as one line in /etc/etminan-verifier/notify-plugins.conf:

pagerduty /etc/etminan-verifier/notify-plugins/pagerduty.sh <sha256sum output>

A stale or wrong hash means the verifier refuses to run the plugin at all — a clear startup error, never a silent bypass. Recompute it every time you touch the file.

Enable, filter, and (optionally) reformat it

In /etc/etminan-verifier/verifier.env:

ETMINAN_NOTIFY_CHANNELS=pagerduty,slack
ETMINAN_PAGERDUTY_ROUTING_KEY=...
ETMINAN_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
# optional: page only on critical findings, let slack see everything
ETMINAN_NOTIFY_PAGERDUTY_SEVERITIES=critical
# optional: reshape this channel's message text — see "What a plugin
# receives" below. Unset falls back to "[{kind}] {host_id}: {text}".
ETMINAN_NOTIFY_SLACK_FORMAT="{severity} on {host_id}: {text} (type: {kind})"

ETMINAN_NOTIFY_CHANNELS is the master switch — unset or empty, no plugin ever executes. Everything else only matters once a channel's name is both listed here and allowlisted in the previous step. A typo in a _SEVERITIES filter can't silently mute a channel: an unknown or empty value (e.g. critcal) is caught by plugins verify and every scheduled run and surfaced as a plugin-failure finding, instead of quietly filtering out every finding so the channel never fires.

Verify before trusting it

$ etminan-verifier plugins verify

Do this right after setup — don't wait for a real finding to discover a typo'd hash or a missing routing key. The same check also runs automatically at the start of every run cycle; a failure there becomes an ordinary warning-severity finding of its own, so a broken notify channel can never fail silently.

04 — What a plugin receives

One JSON array on stdin, never argv.

Finding text is free-form and operator/attacker-influenced — shell-escaping that into a command line is exactly the injection surface this design avoids. Every enabled, allowlisted channel gets the findings from that cycle (already filtered to its own configured severities) as one JSON array on stdin. Nothing on stdout is read; the exit code is the only signal — non-zero means "didn't fire this cycle," logged and skipped, never retried and never allowed to block another channel.

what a notify plugin receives on stdin
[
  {
    "host_id": "web-01",
    "text": "quote signature failed to verify",
    "severity": "critical",
    "kind": "signature-invalid",
    "formatted": "[signature-invalid] web-01: quote signature failed to verify"
  },
  {
    "host_id": "web-02",
    "text": "3 pending item(s) aged past the 24h review SLA",
    "severity": "warning",
    "kind": "sla-exceeded",
    "formatted": "[sla-exceeded] web-02: 3 pending item(s) aged past the 24h review SLA"
  }
]

Five fields, added to in this order as the feature grew — a plugin only reads what it needs, and one written against just the first three keeps working unchanged forever, since every addition here has been purely additive to the array.

FieldWhat it is
host_idThe monitored host this finding is about — or the literal string "verifier" for a finding about the verifier's own configuration (e.g. a broken plugin).
textFree-form, human-readable prose describing exactly what happened.
severity"critical" (a genuine verification failure or operator rejection) or "warning" (escalated, but not the same tier — e.g. a pending review past its SLA).
kindA stable, machine-matchable category — distinct from the free-form text — so a plugin can route or format without ever parsing prose. See the full list below.
formattedThis channel's pre-rendered display text (see "Per-channel formatting" below) — use this instead of hand-assembling host_id/severity/text yourself.

Every kind value

One per distinct failure mode the verifier actually produces — nothing here is speculative, each maps directly to a real code path.

signature-invalid pcr-mismatch unexpected-pcr-selection template-hash-mismatch ak-mismatch nonce-mismatch unreachable not-enrolled check-error sla-exceeded pending-overflow plugin-failure audit-chain-invalid operator-rejected

Per-channel formatting

Set ETMINAN_NOTIFY_<CHANNEL>_FORMAT (channel name uppercased) in verifier.env to change one channel's formatted field — a {kind}/{host_id}/{text}/{severity} template. Purely a config change: no plugin script edit, no re-hashing, no re-allowlisting. Unset falls back to "[{kind}] {host_id}: {text}". Two channels can use two different templates for the exact same finding — and the same knob exists for email too, as ETMINAN_NOTIFY_EMAIL_FORMAT.

05 — Overall layout

The whole message, not just one line — still config-only.

_FORMAT reshapes one finding's line. To change the whole message around those lines — email's subject and any greeting/footer text, Slack's digest header, PagerDuty's payload shape beyond the summary, whether webhook wraps the array — every channel has an optional _TEMPLATE env var pointing at a plain file you write. No plugin script edit, no verifier rebuild, ever. Unset (or unreadable — a loud warning, never a silent failure) means the exact built-in layout every channel has always used.

ChannelEnv varMechanism
EmailETMINAN_NOTIFY_EMAIL_TEMPLATE Plain text — first line Subject: ..., rest is the body.
SlackETMINAN_SLACK_TEMPLATE Plain text — whole digest message.
PagerDutyETMINAN_PAGERDUTY_TEMPLATE A jq filter, not text — see why below.
WebhookETMINAN_WEBHOOK_TEMPLATE A jq filter over the whole array.

Email and Slack: plain text, two placeholders

{count} (how many findings) and {{findings}} (replaced with the joined, already-formatted lines). A template missing {{findings}} gets the lines appended at the end rather than silently dropping them.

example ETMINAN_NOTIFY_EMAIL_TEMPLATE file
Subject: [ALERT] {count} etminan finding(s) need attention

Hi team,

The following {count} issue(s) were detected during the latest check cycle:

{{findings}}

Please review these in `op review`. Thanks,
The Etminan Verifier

PagerDuty and webhook: a jq filter, not text substitution

Deliberate, not an inconsistency: finding text can come straight off an IMA log on a possibly-compromised host and may legally contain quotes or newlines — naive text substitution into a JSON template risks broken or injected JSON on exactly that content. jq's own object/string construction escapes correctly for any content, so these two use a real jq filter instead. Verified directly, not assumed: a finding with an embedded quote, newline, and backslash through the default filter still produces valid, correctly escaped JSON.

PagerDuty's filter sees . as one finding object (host_id/text/severity/kind/formatted) plus $routing_key/$dedup_key as jq variables — a copyable starting point that reproduces the exact built-in default ships at deploy/notify-plugins/ pagerduty-template.jq.example. Webhook's filter sees . as the whole findings array:

example ETMINAN_WEBHOOK_TEMPLATE — wrap the array in an envelope
{source: "etminan-verifier", count: length, findings: .}
06 — Preview

See a rendered result before a real finding ever fires.

$ etminan-verifier notify-preview
$ etminan-verifier notify-preview --channel pagerduty

Renders four representative fake findings (one signature-invalid, one pcr-mismatch, one sla-exceeded, one plugin-failure) through whatever _TEMPLATE/_FORMAT/_SEVERITIES is currently configured, and prints the result: the full email (subject + body) and, per channel in ETMINAN_NOTIFY_CHANNELS (or just --channel <name>), the raw JSON payload that would hit its plugin's stdin.

Never executes a plugin, never calls sendmail — nothing is sent for real, so it's safe to run any time, including against a channel that isn't allowlisted yet (preview never touches the verify-then-exec path at all). This previews the verifier's side of the contract only: what a Slack/PagerDuty/webhook _TEMPLATE does with that payload happens inside the plugin script, which preview deliberately never runs. Pipe the printed payload into the script by hand if you want to see its actual transformed output — that performs the real send, so only do that deliberately.

07 — Test it for real

Confirm a channel actually delivers, not just that it's configured correctly.

$ etminan-verifier notify-test --channel pagerduty

plugins verify already confirms a channel is allowlisted, root-owned, and its pinned hash matches — an integrity/config check, not a functional one. It never executes the plugin, so a channel can pass plugins verify cleanly and still fail the first time it's needed for real: a wrong PagerDuty routing key, an expired Slack webhook URL, an outbound firewall rule blocking the verifier host.

notify-test closes that gap. It builds one synthetic finding (kind: "test", severity: "warning", host_id "notify-test" — distinct from any real monitored host so it can never collide with a real host's own PagerDuty dedup_key) and pipes it through the plugin's real invocation path — the exact same verify-then-exec call a live run cycle uses. Unlike notify-preview, this performs a real send and reports the plugin's actual exit status (and any stderr it printed) back to you. It also deliberately ignores that channel's configured _SEVERITIES filter — you're asking to test this channel right now, not to have the attempt silently skipped because a test finding's severity doesn't happen to match what you've configured for real alerts.

08 — Adding a new channel

A shell script and a hash — that's the whole integration surface.

Microsoft Teams, SMS, an internal on-call tool — any target not shipped is the same recipe as the three above, and needs a verifier rebuild for none of it.

the whole contract, minimal example
#!/bin/sh
set -eu
: "${MY_CHANNEL_URL:?not set}"

findings=$(cat)              # the JSON array, on stdin
text=$(echo "$findings" | jq -r 'map(.formatted) | join("\n")')

curl -fsS --max-time 20 -H "Content-Type: application/json" \
  -X POST "$MY_CHANNEL_URL" -d "$(jq -n --arg t "$text" '{text:$t}')"
                                # exit 0 = fired, non-zero = "skipped this cycle"

Write the script

Read a JSON findings array from stdin, do whatever the target needs, exit non-zero only on a real failure. Copy deploy/notify-plugins/webhook.sh as a starting point — it's kept deliberately minimal for exactly this.

Install it, root-owned, allowlist it, name it

Same two steps as any shipped channel: install to /etc/etminan-verifier/notify-plugins/, add its path and sha256sum to notify-plugins.conf, add its name to ETMINAN_NOTIFY_CHANNELS.

Run plugins verify

Confirms the allowlist entry and the pinned hash both check out, without waiting for a real finding to find out the hard way. Nothing else in the verifier or its wire format needs to change — notify.rs's dispatch, severity filtering, and per-channel formatting are already channel-agnostic.

Full security model (why root-owned, why content-hash pinned, what's re-verified on every single run) lives in the Etminan Plugin API — shared with change-source correlation, since both categories are the same underlying mechanism.