etminan-verifier run always prints every finding and, if
configured, emails it. PagerDuty, Slack, and a generic webhook ship as ready-to-use
channels on top of that — each an external, allowlisted script under the same
Etminan Plugin API as change-source correlation, never a
built-in integration baked into the verifier binary. This page covers the full contract:
what a channel receives, how to reshape its messages without touching its script, and
exactly how little work adding a brand-new channel actually is.
The verifier itself has zero built-in knowledge of PagerDuty's API, Slack's webhook format, or anyone else's paging system. Every channel beyond email is a separately vetted, root-owned, SHA-256-pinned external script — the exact same security model change-source (ticket-system) correlation uses, and for the same reason: a customer should never need a verifier rebuild to point an alert at their own on-call tool.
lines of Rust to add, remove, or reconfigure a notification channel. Every channel —
shipped or custom — is enabled, disabled, filtered, and reformatted entirely through
verifier.env and notify-plugins.conf.
No code change, no rebuild, no redeploy of the verifier binary.
Install only the ones you're actually enabling — each is an independent, individually-allowlisted script.
One Events API v2 event per finding. dedup_key =
"attest-<host_id>" correlates repeat findings for the same host into one
incident instead of paging fresh every cycle. kind rides
along as a custom_details field PagerDuty's own routing
rules can match on.
Needs: ETMINAN_PAGERDUTY_ROUTING_KEY
One digest message for the whole batch — a noisy check cycle with many findings is still a single Slack message, one line per finding, not a flood.
Needs: ETMINAN_SLACK_WEBHOOK_URL
Passes the findings array through unmodified as the POST body — any HTTPS endpoint that accepts JSON, and the template to copy when writing a fully custom channel.
Needs: ETMINAN_WEBHOOK_URL, optional
ETMINAN_WEBHOOK_AUTH_HEADER
etminan-verifier plugins install pagerduty (or
slack/webhook) is designed to do
steps 1 and 2 below at once — it verifies a catalog manifest's Ed25519 signature against a
compiled-in trust anchor before trusting anything it lists, then downloads the
script, hashes it, and writes the allowlist line itself. The sha256 pin still happens
either way; catalog install just adds a second, cryptographic layer in front of it instead
of it being the only thing standing between you and running someone else's script.
Not yet usable in practice: the command and its verification are shipped in
etminan-verifier, but nothing publishes a real, signed catalog
manifest at its default URL yet — see the plugin
directory for current status. Until that exists, this is the only real path:
Add it as one line in
/etc/etminan-verifier/notify-plugins.conf:
A stale or wrong hash means the verifier refuses to run the plugin at all — a clear startup error, never a silent bypass. Recompute it every time you touch the file.
In /etc/etminan-verifier/verifier.env:
ETMINAN_NOTIFY_CHANNELS is the master switch — unset or
empty, no plugin ever executes. Everything else only matters once a channel's name
is both listed here and allowlisted in the previous step. A typo in a
_SEVERITIES filter can't silently mute a channel: an unknown
or empty value (e.g. critcal) is caught by
plugins verify and every scheduled run
and surfaced as a plugin-failure finding, instead of quietly
filtering out every finding so the channel never fires.
Do this right after setup — don't wait for a real finding
to discover a typo'd hash or a missing routing key. The same check also runs
automatically at the start of every run cycle; a
failure there becomes an ordinary warning-severity finding of its own, so a broken
notify channel can never fail silently.
Finding text is free-form and operator/attacker-influenced — shell-escaping that into a command line is exactly the injection surface this design avoids. Every enabled, allowlisted channel gets the findings from that cycle (already filtered to its own configured severities) as one JSON array on stdin. Nothing on stdout is read; the exit code is the only signal — non-zero means "didn't fire this cycle," logged and skipped, never retried and never allowed to block another channel.
[
{
"host_id": "web-01",
"text": "quote signature failed to verify",
"severity": "critical",
"kind": "signature-invalid",
"formatted": "[signature-invalid] web-01: quote signature failed to verify"
},
{
"host_id": "web-02",
"text": "3 pending item(s) aged past the 24h review SLA",
"severity": "warning",
"kind": "sla-exceeded",
"formatted": "[sla-exceeded] web-02: 3 pending item(s) aged past the 24h review SLA"
}
]
Five fields, added to in this order as the feature grew — a plugin only reads what it needs, and one written against just the first three keeps working unchanged forever, since every addition here has been purely additive to the array.
| Field | What it is |
|---|---|
| host_id | The monitored host this finding is about — or the literal string
"verifier" for a finding about the verifier's own
configuration (e.g. a broken plugin). |
| text | Free-form, human-readable prose describing exactly what happened. |
| severity | "critical" (a genuine verification
failure or operator rejection) or "warning" (escalated, but
not the same tier — e.g. a pending review past its SLA). |
| kind | A stable, machine-matchable category — distinct from the free-form
text — so a plugin can route or format without ever parsing
prose. See the full list below. |
| formatted | This channel's pre-rendered display text (see
"Per-channel formatting" below) — use this instead of hand-assembling
host_id/severity/text yourself. |
kind valueOne per distinct failure mode the verifier actually produces — nothing here is speculative, each maps directly to a real code path.
Set ETMINAN_NOTIFY_<CHANNEL>_FORMAT (channel name
uppercased) in verifier.env to change one channel's
formatted field — a {kind}/{host_id}/{text}/{severity} template. Purely a config change: no plugin script edit,
no re-hashing, no re-allowlisting. Unset falls back to
"[{kind}] {host_id}: {text}". Two channels can use two
different templates for the exact same finding — and the same knob exists for email too,
as ETMINAN_NOTIFY_EMAIL_FORMAT.
_FORMAT reshapes one finding's line. To change the
whole message around those lines — email's subject and any greeting/footer text,
Slack's digest header, PagerDuty's payload shape beyond the summary, whether webhook wraps
the array — every channel has an optional _TEMPLATE env var
pointing at a plain file you write. No plugin script edit, no verifier rebuild, ever.
Unset (or unreadable — a loud warning, never a silent failure) means the exact built-in
layout every channel has always used.
| Channel | Env var | Mechanism |
|---|---|---|
ETMINAN_NOTIFY_EMAIL_TEMPLATE |
Plain text — first line Subject: ..., rest is the body. | |
| Slack | ETMINAN_SLACK_TEMPLATE |
Plain text — whole digest message. |
| PagerDuty | ETMINAN_PAGERDUTY_TEMPLATE |
A jq filter, not text — see why below. |
| Webhook | ETMINAN_WEBHOOK_TEMPLATE |
A jq filter over the whole array. |
{count} (how many findings) and {{findings}} (replaced with the joined, already-formatted lines). A template missing {{findings}} gets the lines appended at the end rather than silently
dropping them.
Subject: [ALERT] {count} etminan finding(s) need attention
Hi team,
The following {count} issue(s) were detected during the latest check cycle:
{{findings}}
Please review these in `op review`. Thanks,
The Etminan Verifier
Deliberate, not an inconsistency: finding text can come straight off an IMA log on a possibly-compromised host and may legally contain quotes or newlines — naive text substitution into a JSON template risks broken or injected JSON on exactly that content. jq's own object/string construction escapes correctly for any content, so these two use a real jq filter instead. Verified directly, not assumed: a finding with an embedded quote, newline, and backslash through the default filter still produces valid, correctly escaped JSON.
PagerDuty's filter sees . as one finding object
(host_id/text/severity/kind/formatted) plus $routing_key/$dedup_key as jq variables — a copyable starting point that
reproduces the exact built-in default ships at deploy/notify-plugins/
pagerduty-template.jq.example. Webhook's filter sees . as the whole findings array:
{source: "etminan-verifier", count: length, findings: .}
Renders four representative fake findings (one signature-invalid, one pcr-mismatch, one
sla-exceeded, one plugin-failure) through whatever _TEMPLATE/_FORMAT/_SEVERITIES is currently
configured, and prints the result: the full email (subject + body) and, per channel in
ETMINAN_NOTIFY_CHANNELS (or just --channel <name>), the raw JSON payload that would hit its
plugin's stdin.
Never executes a plugin, never calls sendmail — nothing is sent for real, so it's
safe to run any time, including against a channel that isn't allowlisted yet (preview never
touches the verify-then-exec path at all). This previews the verifier's side of the
contract only: what a Slack/PagerDuty/webhook _TEMPLATE does
with that payload happens inside the plugin script, which preview deliberately never runs.
Pipe the printed payload into the script by hand if you want to see its actual transformed
output — that performs the real send, so only do that deliberately.
plugins verify already confirms a channel is allowlisted,
root-owned, and its pinned hash matches — an integrity/config check, not a functional one.
It never executes the plugin, so a channel can pass plugins verify
cleanly and still fail the first time it's needed for real: a wrong PagerDuty routing key, an
expired Slack webhook URL, an outbound firewall rule blocking the verifier host.
notify-test closes that gap. It builds one synthetic finding
(kind: "test", severity: "warning",
host_id "notify-test" — distinct from any real monitored host so
it can never collide with a real host's own PagerDuty dedup_key)
and pipes it through the plugin's real invocation path — the exact same
verify-then-exec call a live run cycle uses. Unlike
notify-preview, this performs a real send and reports the
plugin's actual exit status (and any stderr it printed) back to you. It also deliberately
ignores that channel's configured _SEVERITIES filter — you're
asking to test this channel right now, not to have the attempt silently skipped because a
test finding's severity doesn't happen to match what you've configured for real alerts.
Microsoft Teams, SMS, an internal on-call tool — any target not shipped is the same recipe as the three above, and needs a verifier rebuild for none of it.
#!/bin/sh
set -eu
: "${MY_CHANNEL_URL:?not set}"
findings=$(cat) # the JSON array, on stdin
text=$(echo "$findings" | jq -r 'map(.formatted) | join("\n")')
curl -fsS --max-time 20 -H "Content-Type: application/json" \
-X POST "$MY_CHANNEL_URL" -d "$(jq -n --arg t "$text" '{text:$t}')"
# exit 0 = fired, non-zero = "skipped this cycle"
Read a JSON findings array from stdin, do whatever the target needs, exit non-zero
only on a real failure. Copy deploy/notify-plugins/webhook.sh
as a starting point — it's kept deliberately minimal for exactly this.
Same two steps as any shipped channel: install to
/etc/etminan-verifier/notify-plugins/, add its path and
sha256sum to notify-plugins.conf,
add its name to ETMINAN_NOTIFY_CHANNELS.
plugins verifyConfirms the allowlist entry and the pinned hash both check out, without waiting for
a real finding to find out the hard way. Nothing else in the verifier or its wire
format needs to change — notify.rs's dispatch, severity
filtering, and per-channel formatting are already channel-agnostic.
Full security model (why root-owned, why content-hash pinned, what's re-verified on every single run) lives in the Etminan Plugin API — shared with change-source correlation, since both categories are the same underlying mechanism.