Etminan integrates with things it can't build support for itself — whatever change-management or ticket system a customer already runs, and whatever paging/chat/ webhook target their on-call process already uses. Rather than hardcoding a fixed list of integrations, every one of these is a plugin: a separately-vetted external script or executable, in any language, that satisfies a documented, category-specific contract. This page is that contract, with a full worked example.
A customer's ticket system is Jira, ServiceNow, GitLab Issues, Request Tracker, or something built in-house — and their on-call process already pages through PagerDuty, Slack, or a webhook into some other tool. Etminan will never support all of these as first-party Rust code, and it shouldn't try to: every hardcoded integration is a permanent maintenance burden and an HTTP client the verifier didn't need. Instead, adding an integration is: write a script, allowlist it (name, path, pinned content hash), reference its name in one environment variable. No Rust code, no verifier rebuild, ever — true today and stays true as more categories are added later.
Implemented once, in verifier/src/plugin_exec.rs, shared by
every plugin category so this is never re-implemented — or accidentally weakened — per
category. Before executing any plugin, every time, never cached from a prior run:
Refuses a symlink swapped in after any earlier check — never re-resolves the path a second time before exec, closing the window a naive check-then-exec would leave.
Must be a regular file, owned by uid 0, not group- or world-writable — the same bar
sudo/SSH already apply to a config or key file.
Hashes that exact open file and compares it against the pinned SHA-256 in the category's allowlist file. A modified plugin — accidentally or maliciously — is refused, not silently run with different behavior.
Runs that same already-verified file descriptor via
/proc/self/fd/<n>, under a bounded timeout — an
unresponsive plugin can't hang a whole check/run cycle — as a separate,
unprivileged ETMINAN_PLUGIN_USER (default
etminan-verifier-plugin), not sharing
etminan-verifier's own user. The first three steps prove a
plugin's bytes are exactly what was approved — integrity,
not that an approved plugin is safe to run. A maliciously authored plugin
that passes every check above still shouldn't be able to read
baseline.db or the operator's signing key just because it
was allowed to execute — this is execution sandboxing, closing that
gap. Requires CAP_SETUID/CAP_SETGID
(granted via the systemd unit's AmbientCapabilities=);
invoked outside that unit, this degrades automatically to running as
etminan-verifier's own user, with a warning, rather than
breaking every interactive invocation.
Any failure at any of these steps — missing file, wrong owner, writable by someone else,
hash mismatch, timeout, non-zero exit, malformed output — is treated as "this plugin didn't
check out": logged, and excluded from that cycle's results. It is never a crash, and it
must never be silently ignored either: every category feeds its verification
outcome into etminan-verifier plugins verify (an on-demand
check) and into run's automatic per-cycle check, turning a
broken plugin into an ordinary, always-visible finding — not a line in stderr someone has
to be watching for.
A category's allowlist file (change-source-plugins.conf,
notify-plugins.conf, and any future category's own file) is
never a directory scan: a plugin that isn't listed there by name is never
run, no matter what's on disk.
A plugin's output is validated too, not just its execution. Stdout/stderr
are capped at 1 MiB — a resource-exhaustion plugin can't grow
etminan-verifier's memory without limit. Every free-form field a
plugin returns (a change-source's id/summary/status, a notify channel's
rendered formatted text) has control characters — what an
ANSI/terminal-escape sequence is built from — stripped and its length bounded
before it's stored or displayed, so a malicious plugin can't use op review's terminal output, an email body, or another
channel's payload as an injection vector. A url field is kept
only if it's a plausible http(s):// link with no control
characters or whitespace — dropped otherwise.
Every check above runs regardless of how a plugin's hash got into the allowlist file in the
first place — by default, that's an operator computing sha256sum
by hand and copying it in, which is correct but pure trust-on-first-use. etminan-verifier plugins install/update <name> (verifier/src/plugin_catalog.rs) adds a second, cryptographic layer in
front of that: it fetches a catalog manifest, verifies a detached Ed25519
signature over it against a trust anchor compiled into the
etminan-verifier binary itself — no keyring to import, no
subprocess call to gpg — and only then trusts the SHA-256 it
lists for that plugin, downloads it, and writes the allowlist line. The signing key is a
dedicated, single-purpose Ed25519 keypair used for nothing else — not the GPG key that signs
release tarballs, not any operator's own signing key — so compromising it only ever lets an
attacker forge catalog entries, never reach baseline approvals, the audit log, or a release.
Not yet live in practice: this verification code and the trust anchor are shipped, but
nothing yet publishes a real, signed catalog manifest for it to fetch — see the
plugin directory for current status. Until that exists,
hand-pinning the SHA-256 yourself (above) is the only path, and remains fully supported
either way — the catalog is an added convenience and an added guarantee, never a
replacement for it.
etminan#!/bin/sh, an ELF binary, etc.). The verifier never
cares what's inside, only what the file hashes to.env_clear) and passes the plugin only the
ETMINAN_* namespace plus basic system vars (PATH,
HOME, LANG/LC_*, TZ, TMPDIR,
TERM, USER, LOGNAME, SHELL) — so pick a
distinct ETMINAN_<X>_* prefix for your plugin's config. Anything the
verifier happens to run with outside that set never reaches the plugin, so an unrelated
secret can't leak to an approved-but-later-compromised plugin.0 means it ran successfully; non-zero means "didn't do its
job this cycle," with a clear message on stderr. A plugin must never print a partial or
best-guess result and exit 0.| Category | Input | Output | Reference plugin(s) |
|---|---|---|---|
| change-source | argv: --host <id> --anchor <RFC3339 timestamp> |
JSON array on stdout: [{"id","summary","status","url"}, ...],
[] for "checked, nothing found" |
deploy/change-sources/{request-tracker,freeitsm}.sh |
| notify | JSON array on stdin: [{"host_id","text","severity","kind","formatted"}, ...] |
nothing read from stdout; exit code is the only signal | deploy/notify-plugins/{pagerduty,slack,webhook}.sh |
Each category has its own full walkthrough: change-source correlation and notification channels. A future category adds a row to this table and its own short doc — the security model and invocation shape above don't change.
Why change-source takes structured input via argv but
notify takes it via stdin: a change-source plugin's whole job
is to go query an external system using the host/time it's given, so a couple of
scalar values as flags is natural and lets a plugin be tested by hand trivially. A notify
plugin's input is a list of findings whose text is free-form and
attacker/operator-influenced — passing that through argv would mean shell-escaping
arbitrary text into command-line arguments, exactly the kind of injection surface this
project avoids elsewhere. JSON on stdin sidesteps that entirely.
An abridged version of the reference plugin shipped at
deploy/notify-plugins/slack.sh — the real script additionally
uses each finding's pre-rendered formatted field, escapes Slack
mrkdwn control characters (&/</>)
in the message text, and supports an optional ETMINAN_SLACK_TEMPLATE
whole-message layout. It satisfies the notify category's
contract exactly: no arguments, findings arrive as JSON on stdin, config comes from the
environment, exit code is the only signal read back.
#!/bin/sh
# Reads ETMINAN_SLACK_WEBHOOK_URL from the inherited environment —
# never from argv. Requires curl and jq.
set -eu
: "${ETMINAN_SLACK_WEBHOOK_URL:?ETMINAN_SLACK_WEBHOOK_URL not set}"
# the findings JSON array arrives whole, on stdin
findings=$(cat)
# one digest message per invocation, not one per finding
text=$(echo "$findings" | jq -r '
["etminan-verifier: " + (length | tostring) + " finding(s):"]
+ (map(" [" + .host_id + "] (" + .severity + ") " + .text))
| join("\n")
')
payload=$(jq -n --arg text "$text" '{text: $text}')
curl -fsS --max-time 20 \
-H "Content-Type: application/json" \
-X POST "$ETMINAN_SLACK_WEBHOOK_URL" \
-d "$payload" >/dev/nullNote what it deliberately does not do: no argument parsing (the contract for this category says none), no retry loop (a non-zero exit is the verifier's cue to log "this channel didn't fire this cycle" and move on, never to block or retry), and one Slack message per invocation rather than one per finding — a noisy cycle with many findings should still be a single message, not a flood.
Add one line to /etc/etminan-verifier/notify-plugins.conf —
name, path, pinned hash, whitespace-separated:
In /etc/etminan-verifier/verifier.env:
Confirms the plugin checks out without waiting for a real
finding to fire it — this same check also runs automatically at the start of every
run cycle.
Restart is never required for a config-only change — etminan-verifier
is invoked fresh per op check/run, not
a long-lived daemon reading env vars once at startup, so the next scheduled
run (or a manual plugins verify)
picks up new config immediately.
Read its row in the table above, and its own doc page for a worked example.
Satisfy that category's input/output contract, and read whatever config it needs
from its own environment — pick a distinct ETMINAN_<X>_*
prefix so it doesn't collide with another configured plugin.
Somewhere not group/world-writable, e.g.
/etc/etminan-verifier/notify-plugins/my-plugin.sh.
In that category's .conf file:
<name> <path> <sha256sum output>.
Recompute the hash every time you touch the file.
Add its name to that category's enable-list environment variable
(ETMINAN_CHANGE_SOURCES or
ETMINAN_NOTIFY_CHANNELS).
Confirms the plugin is allowlisted and passes every check above, without waiting for a real check/run cycle to find out the hard way.
Steps 3–5 above (download, hash it yourself, hand-copy the hash into a
.conf file) are the manual path — always available, and the
only path for a plugin you or a third party wrote yourselves. For a reference plugin
Etminan itself publishes and vouches for, plugins install
collapses those into one command, and "installed via this command" is itself the guarantee
that the plugin is genuinely Etminan's, unmodified, and safe to run:
plugins install verifies the whole catalog's Ed25519 signature
against a dedicated, single-purpose key compiled into the etminan-verifier binary (not a GPG key, not any operator's key) before
trusting a single entry in it, then downloads the plugin, re-verifies its content hash
against the catalog's pinned value, writes it into
/etc/etminan-verifier/<type>-plugins/, and appends the
correct .conf entry — exactly the file this page's own
verify-then-exec model already checks at execution time, so a catalog-installed plugin is
held to the identical bar as a hand-installed one. It must run as root, for the same reason
step 3 above says to install a plugin root-owned in the first place.
It deliberately stops short of one thing: it never edits
ETMINAN_CHANGE_SOURCES/ETMINAN_NOTIFY_CHANNELS
for you, since those live in the verifier's own systemd unit environment, not a file this
command manages — it prints the exact line to add and reminds you to restart
etminan-verifier.service, step 5 above, still a deliberate,
explicit action. plugins update <name> re-checks the
catalog and re-pins an already-installed plugin to whatever version it currently lists —
only when you run it, never automatically or in the background.
Not yet usable in practice: the client side above is real and shipped, but nothing
publishes a real, signed plugins-catalog.json at its default
URL yet — see the Plugin Directory for current status
per plugin. Until that exists, the manual six-step path above remains the only real route.
Checks every plugin currently referenced by any category's enable-list variable: confirms it has an allowlist entry and passes the full security check above (ownership, permissions, content hash) — without executing it. Exits non-zero and prints exactly what's wrong if anything fails.
This is also run automatically, silently-in-the-success-case, at the start of every
run cycle — a failure there is turned into an ordinary
warning-severity finding (resource "verifier", not a real host
id) that goes through the same print/email/notify/audit-log path as any other finding,
specifically so a broken notification channel doesn't get to hide its own
breakage: if even one channel still works, or the local structured log/print output, the
problem is visible.