# Notification channels — PagerDuty, Slack, generic webhook

This walks through configuring additional alarm-notification channels end
to end. For the architecture and design rationale, see `docs/design.md`'s
Alarm path section. For the shared plugin contract this feature
implements, see [`docs/plugins.md`](plugins.md) — this page is the
`notify` category's worked example, the same relationship
[`docs/change-sources.md`](change-sources.md) has to that shared spec.

## What this is

`etminan-verifier run` already prints and, if `ETMINAN_NOTIFY_EMAIL` is
set, emails every finding from a check cycle. This feature adds more
channels — PagerDuty, Slack, and a generic webhook are shipped as
reference plugins, and any channel a customer needs (Microsoft Teams, SMS,
an internal on-call system) is a script away, never a Rust change. Every
channel — email included, conceptually, though email itself predates this
feature and stays inline in `alarm.rs` rather than becoming a plugin — is
independently enabled and independently filterable by severity, so e.g.
`critical` findings can page while `warning`-tier SLA breaches only email.

This project has no in-process HTTP client at all (`ureq` was removed
from the verifier entirely in 0.5.0, specifically to avoid that
dependency's own trust/timeout surface) — every notify channel is an
**external plugin process**, same model and same security guarantees as
change-source correlation. See [`docs/plugins.md`](plugins.md)'s security
model section for exactly what's verified before a plugin ever runs.

## Prerequisites

- The shipped reference plugins (`deploy/notify-plugins/*.sh`) need
  `curl`; `pagerduty.sh` and `slack.sh` also need `jq`.
- Whatever credential/URL the channel you're enabling needs:
  - **PagerDuty**: an Events API v2 integration routing key (Service →
    Integrations → Add Integration → Events API v2 in PagerDuty's UI).
  - **Slack**: an incoming webhook URL (Slack app config →
    Incoming Webhooks).
  - **Generic webhook**: any HTTPS endpoint that accepts a JSON POST, plus
    an optional auth header if it requires one.

## Configuration

**1. Install the plugin script(s)** somewhere root-owned and not
group/world-writable — e.g.:

```bash
sudo install -D -m 0755 -o root -g root \
  deploy/notify-plugins/pagerduty.sh \
  /etc/etminan-verifier/notify-plugins/pagerduty.sh
```

Repeat for `slack.sh`/`webhook.sh` as needed — only install the channels
you're actually enabling.

**2. Allowlist it/them**, with each one's exact content hash. Copy
`deploy/notify-plugins.conf.example` to
`/etc/etminan-verifier/notify-plugins.conf` and uncomment/add the
channels you installed:

```
pagerduty /etc/etminan-verifier/notify-plugins/pagerduty.sh <sha256sum output>
slack     /etc/etminan-verifier/notify-plugins/slack.sh     <sha256sum output>
```

```bash
sha256sum /etc/etminan-verifier/notify-plugins/pagerduty.sh
```

**Recompute and update that hash every time you touch a plugin file** —
a stale/wrong hash means the verifier refuses to run it at all (a clear
error, not a silent skip), never a security bypass.

**3. Configure each channel**, in `/etc/etminan-verifier/verifier.env`
(see `deploy/verifier.env.example` for the shipped, commented-out
template). These are read by the *plugin*, not the verifier binary:

```bash
ETMINAN_NOTIFY_CHANNELS=pagerduty,slack

ETMINAN_PAGERDUTY_ROUTING_KEY=...
ETMINAN_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...

# Optional: page only on critical findings, let slack see everything.
ETMINAN_NOTIFY_PAGERDUTY_SEVERITIES=critical
```

`ETMINAN_NOTIFY_CHANNELS` is the master switch: unset or empty, this
feature never executes any plugin at all. Leaving it unset is always
safe — every other setting only matters once a channel's name is both
listed here *and* allowlisted in step 2.

**4. Verify it**:

```bash
etminan-verifier plugins verify
```

Confirms every channel in `ETMINAN_NOTIFY_CHANNELS` is allowlisted and
passes verification — do this right after setup, don't wait for a real
finding to discover a typo'd hash or a missing routing key. This same
check also runs automatically at the start of every `run` cycle; a
failure there becomes an ordinary `warning`-severity finding (so a broken
notify channel can never fail silently — see `docs/plugins.md`'s
"Verifying your configuration" section for why).

Restart is not required for a config-only change — `etminan-verifier` is
invoked fresh per `check`/`run`, so the next scheduled `run` picks up new
config immediately.

## What a plugin receives

Every enabled, allowlisted channel gets the full list of findings from
that `run` cycle (filtered to its own configured severities first) as a
JSON array on stdin:

```json
[
  {"host_id": "web-01", "text": "quote signature invalid", "severity": "critical"},
  {"host_id": "web-02", "text": "3 pending item(s) aged past the 24h review SLA", "severity": "warning"}
]
```

`pagerduty.sh` sends one Events API v2 event per finding (`dedup_key`
`attest-<host_id>`, so repeat findings for the same host correlate into
one incident rather than paging fresh every cycle). `slack.sh` sends one
digest message for the whole batch. `webhook.sh` passes the array through
unmodified — the template to copy for a fully custom integration.

## Adding a new channel

No Rust code, no verifier rebuild — see
[`docs/plugins.md`](plugins.md#writing-a-new-plugin) for the full
walkthrough (shared with change-source plugins). The short version for
this category specifically: write a script that reads a JSON findings
array from stdin and exits non-zero on failure, install it, allowlist it
in `notify-plugins.conf`, add its name to `ETMINAN_NOTIFY_CHANNELS`, and
run `etminan-verifier plugins verify`.