etminan
Backup & disaster recovery

Back up the one host you can't re-derive.

Almost all of an Etminan deployment is re-installable from the package and re-derivable by re-enrolling. Exactly one thing is not: the verifier trust store — the baselines, the hash-chained audit log, the pinned host fingerprints, and this verifier's mTLS identity. Lose it and you re-enroll every monitored host from scratch. This page is the disaster-recovery plan for that host, plus three ready-to-adapt example files for Bacula.

!

These three files are a Bacula example — but the backup itself is required. They are what a correct disaster-recovery backup of an Etminan deployment needs to capture. The files are written for Bacula specifically (FileSets, Jobs, a ClientRunBeforeJob snapshot).

If you use a different backup solution — restic, Borg, Veeam, Duplicati, bareos, a filesystem snapshot, anything — you must adapt them accordingly: the file formats won't apply, but the three responsibilities they encode (a WAL-safe DB snapshot, the trust-store file set — which now includes the RBAC daemon's signing key — and locking down restore access to that key) still do. None of these files is installed by the package; they are examples you copy onto your backup infrastructure.

01 — What must survive a disaster

One host is catastrophic. The rest is convenience.

Back up by blast radius, not by host count. The table is the whole priority order.

DataWhereIf lostPriority
Verifier trust store the verifier host — baseline.db (SQLite/WAL), its audit-chain anchor, state.json, tls/, /etc/etminan-verifier Catastrophic. Baselines, audit history and every pinned host fingerprint are gone; re-enroll every host and rebuild every baseline. Must back up
RBAC daemon signing key on the verifier host — /var/lib/etminan-verifier/daemon-signing.key (0600, owned by the daemon user). Operators hold no key of their own. Cannot re-verify existing signed baseline batches. It lives on the verifier by design, so it is captured by the verifier trust-store backup. Part of the trust store
Agent state each monitored host — tls/, assigned_profile, /etc/etminan-agent Re-derivable: re-enroll or op rotate-tls from the verifier. Backed up only to make restore a copy instead of a re-enrollment. Optional / nice
TPM AK/EK key material inside each host's TPM Not backup-able by design. A TPM clear forces re-enrollment of that host regardless of any backup. Cannot back up
!

The verifier DB is SQLite in WAL mode. A naive cp of a live baseline.db yields a stale or corrupt restore, and its external audit-chain anchor can end up ahead of the copied DB — which the verifier reports as tampering. That is the entire reason for the snapshot pre-job below, and the reason every backup tool needs adapting rather than pointing at the raw file.

02 — The three files

Copy these onto your Bacula infrastructure, then adapt the paths.

Two go on the Director; one runs on the verifier host. Each is heavily commented — the comments are the documentation. Download them here and read the full source below.

Bacula Director

etminan-filesets.conf

What to back up. Defines the Verifier and Agent FileSets — the verifier one includes the daemon signing key and excludes the live SQLite DB in favour of the snapshot.

Include first, before the Jobs file.

etminan-filesets.conf
Bacula Director

etminan-jobs.conf

When and how. Schedules, Pools, JobDefs and Backup/Restore Jobs that reference those FileSets, wire in the pre-job, and split restore access with Console ACLs.

Companion to the FileSets file.

etminan-jobs.conf
Verifier host

etminan-verifier-snapshot.sh

The ClientRunBeforeJob that produces a WAL-safe .backup snapshot of baseline.db plus a matching audit-chain anchor — zero race window by construction.

Needs: sqlite3

etminan-verifier-snapshot.sh
03 — FileSets

etminan-filesets.conf — what to back up

etminan-filesets.conf

Two FileSets: the catastrophic verifier trust store (backing up the snapshot, never the live DB, and including the RBAC daemon's signing key) and the mostly-re-derivable per-host agent state. There is no separate operator-keys set — in the daemon model operators hold no key.

etminan-filesets.conf
# Bacula FileSets for Etminan disaster recovery.
#
# EXAMPLE / REFERENCE ONLY — this is not installed by the etminan package and is
# not part of the deliverable. Copy it onto your Bacula Director and adapt the
# paths and the "<-- site resource" names to your own deployment.
#
# A generic backup config for an Etminan deployment — Etminan has exactly two
# installable parts, so there are two installation FileSets:
#   * Etminan-Verifier-FileSet — the verifier install (the trust store; the only
#                                catastrophic-loss host). Includes the RBAC
#                                daemon's signing key — see the note below.
#   * Etminan-Agent-FileSet    — an agent install, per monitored host; mostly
#                                re-derivable. Cloned per host in the Jobs file.
#
# NO separate operator-keys FileSet. In the daemon (op) model, operators hold NO
# signing key at all: the etminan-verifierd daemon holds a single signing key on
# the verifier host and signs each authorised action on the operator's behalf,
# and operator identities are UID->role/scope rows inside baseline.db. So the
# only signing key to back up lives on the verifier and is captured by the
# Verifier FileSet below — there is no off-host key store to protect separately.
#
# NOTE: nothing here is created by any install script — an Etminan deployment is
# installed from the package; the operator hand-creates the real config, TLS
# identity, and (on the verifier) the DB/state. That operator-created state is
# exactly what these FileSets capture, since the package ships only *.example
# templates and empty directories.
#
# IMPORTANT — pair the verifier FileSet with the snapshot pre-job. baseline.db
# is SQLite/WAL; this FileSet backs up the *snapshot* produced by
# etminan-verifier-snapshot.sh, never the live DB. The Jobs, Schedules, Pools,
# and the ClientRunBeforeJob wiring live in the companion etminan-jobs.conf.
# Include both from your Director config, FileSets first:
#
#   @/etc/bacula/etminan/etminan-filesets.conf   # example path — put them
#   @/etc/bacula/etminan/etminan-jobs.conf        # wherever your Director reads

FileSet {
  Name = "Etminan-Verifier-FileSet"
  Include {
    Options {
      signature   = SHA256
      compression = GZIP9
      # Never back up the live SQLite DB or its sidecars — the snapshot below
      # is the consistent copy. Excluding these avoids a corrupt/stale restore.
      WildFile = "/var/lib/etminan-verifier/baseline.db"
      WildFile = "/var/lib/etminan-verifier/baseline.db-wal"
      WildFile = "/var/lib/etminan-verifier/baseline.db-shm"
      WildFile = "/var/lib/etminan-verifier/baseline.db.audit-head"
      Exclude  = yes
    }

    Options {
      signature   = SHA256
      compression = GZIP9
    }

    # --- The trust store (CATASTROPHIC if lost) ---------------------------
    # WAL-safe SQLite snapshot + its consistent audit-chain anchor, produced by
    # ClientRunBeforeJob = etminan-verifier-snapshot.sh. Restore: copy
    # baseline.db.snapshot -> baseline.db and baseline.db.snapshot.audit-head ->
    # baseline.db.audit-head (see the exclude block above — never the live DB).
    # The snapshot also carries the operator identity registry (UID->role/scope)
    # and all signed baseline batches, since both live inside baseline.db.
    File = /var/backups/etminan/baseline.db.snapshot
    File = /var/backups/etminan/baseline.db.snapshot.audit-head

    # The RBAC daemon's signing key (CROWN JEWEL). etminan-verifierd holds this
    # single Ed25519 seed and signs every authorised operator action with it;
    # operators hold no key of their own. It lives on the verifier host by
    # design, so it is part of this trust store — restore it together with the
    # DB snapshot so signed batches keep verifying. 0600, owned by the daemon
    # user: preserve those perms/owner on restore (see the Console ACL note in
    # etminan-jobs.conf — only the DR role can restore this pool).
    File = /var/lib/etminan-verifier/daemon-signing.key

    # Per-host enrollment + replay state: pinned AK/EK/TLS fingerprints, PCR10
    # cursor, signed enrollment/rotation accountability. Written atomically
    # (temp+fsync+rename) so a live copy is always consistent — safe as-is.
    # Losing it forces re-enrollment of every host from scratch.
    File = /var/lib/etminan-verifier/state.json

    # This verifier's mTLS identity. Its fingerprint is pinned into every agent
    # (ETMINAN_VERIFIER_CERT_FINGERPRINT). Losing it forces keygen-tls + re-pin
    # on every agent by hand.
    File = /var/lib/etminan-verifier/tls

    # --- Config + plugin allowlists (recoverable, but back up) ------------
    # verifier.env holds plaintext integration secrets (RT/FreeITSM/PagerDuty/
    # Slack/webhook). The two *-plugins.conf files carry the SHA256 pins that
    # authorize plugin execution; the *-plugins/ dirs hold the pinned binaries.
    # The conf pins and the binaries must restore together.
    File = /etc/etminan-verifier

    # --- Convenience: fast-restore binary (re-installable from package) ---
    WildFile = "/usr/bin/etminan-verifier"
    File = /usr/bin

    # --- NOT captured here (out of band): -------------------------------------
    #   * Plugin-catalog signing key (~/.etminan-catalog-signing.key) — offline,
    #     never on a verifier host (HANDBOOK §5). Only exists if you self-sign
    #     your own private plugin catalog; most deployments consume the vendor
    #     catalog, whose private half never touches a customer host at all.
    #   * The host TPM is the irreplaceable factor for AGENTS: a TPM clear/reset
    #     invalidates the AK/EK pinned in state.json and forces re-enroll. TPM
    #     key material cannot be backed up.
  }
}

FileSet {
  Name = "Etminan-Agent-FileSet"
  Include {
    Options {
      signature   = SHA256
      compression = GZIP9
    }
    # Agent mTLS identity — fingerprint is pinned in the verifier's state.json.
    # Loss requires re-enroll or op rotate-tls (verifier-side action), not fatal.
    File = /var/lib/etminan-agent/tls

    # Last profile pushed by op assign-profile; re-derivable (authoritative record
    # is in the verifier's state.json + audit_log), included for a clean restore.
    File = /var/lib/etminan-agent/assigned_profile

    # agent.env (holds ETMINAN_VERIFIER_CERT_FINGERPRINT, the pin) + profiles.conf
    File = /etc/etminan-agent

    WildFile = "/usr/bin/etminan-agent"
    File = /usr/bin
    # NOTE: TPM AK/EK are in-TPM only — nothing on disk to back up. A TPM clear
    # forces re-enrollment regardless of this backup.
  }
}
04 — Jobs

etminan-jobs.conf — when & how

etminan-jobs.conf

Schedules, Pools, JobDefs and Jobs. The verifier JobDef wires the snapshot in as Client Run Before Job; separate pools plus two Console ACL roles keep day-to-day agent operators away from the verifier pool — only the DR role can restore the trust store, and therefore the daemon signing key.

etminan-jobs.conf
# Bacula Jobs for Etminan disaster recovery.
#
# EXAMPLE / REFERENCE ONLY — not installed by the etminan package, not part of
# the deliverable. Copy onto your Bacula Director and adapt to your deployment.
#
# This is the companion to etminan-filesets.conf — that file defines *what* to
# back up (Etminan-Verifier-FileSet and Etminan-Agent-FileSet); this file
# defines *when* and *how* (Schedules, Pools, JobDefs, and the Backup/Restore
# Jobs that reference those FileSets). Without this file the FileSets are never
# exercised.
#
# Include both from your Director config, FileSets first:
#
#   @/etc/bacula/etminan/etminan-filesets.conf   # example path — put them
#   @/etc/bacula/etminan/etminan-jobs.conf        # wherever your Director reads
#
# The Director, Catalog, Messages, Client (FileDaemon), and Storage resources
# are site-specific and NOT defined here — supply them from your environment.
# The names referenced below and marked "<-- site resource" must match yours:
#
#   Client   = etminan-verifier-fd , etminan-agent-fd
#   Storage  = File
#   Messages = Standard
#   Catalog  = MyCatalog
#
# There is NO operator-workstation client. In the daemon (op) model, operators
# hold no signing key — the etminan-verifierd daemon holds the single signing
# key on the verifier host, so it is captured by the verifier backup, not a
# separate off-host job (see etminan-filesets.conf).
#
# See Bacula-Forgejo/docs/bacula-director.conf.example for a complete Director/
# Catalog/Storage/Messages scaffold to adapt.

# ── Schedules ─────────────────────────────────────────────────────────────────
# Weekly Full + daily Incremental. The verifier snapshot pre-job regenerates
# baseline.db.snapshot on every run, so Incrementals reliably capture DB changes.

Schedule {
  Name = "EtminanVerifierSchedule"
  Run = Full        1st sun at 01:00
  Run = Incremental mon-sat at 01:00
}

Schedule {
  Name = "EtminanAgentSchedule"
  # Agent state is small and mostly re-derivable — a nightly Full is cheap and
  # keeps restore dead-simple (no chain to replay).
  Run = Full daily at 02:00
}

# ── Pools ─────────────────────────────────────────────────────────────────────
# The verifier trust store is the only catastrophic-loss data AND contains
# plaintext secrets (verifier.env), this verifier's mTLS material, and the RBAC
# daemon's signing key. It gets its own restricted pool so restore access can be
# locked down via Console ACLs below — whoever can restore this pool can restore
# the daemon signing key, so keep that role tight.

Pool {
  Name             = EtminanVerifierPool
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 365 days
  Maximum Volume Bytes = 20G
  Label Format     = "EtminanVerifierVol-"
}

Pool {
  Name             = EtminanAgentPool
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 180 days
  Maximum Volume Bytes = 20G
  Label Format     = "EtminanAgentVol-"
}

# ── JobDefs ───────────────────────────────────────────────────────────────────

JobDefs {
  Name      = "EtminanVerifierDefs"
  Type      = Backup
  Level     = Incremental
  Client    = etminan-verifier-fd          # <-- site resource
  FileSet   = "Etminan-Verifier-FileSet"   # from etminan-filesets.conf
  Storage   = File                         # <-- site resource
  Messages  = Standard                     # <-- site resource
  Pool      = EtminanVerifierPool
  Schedule  = "EtminanVerifierSchedule"
  Priority  = 10
  # CRITICAL: take the WAL-safe snapshot BEFORE the FileSet is read, so the
  # backup captures baseline.db.snapshot (+ .audit-head) instead of the live
  # SQLite DB. The FileSet excludes the live DB; this produces the copy it wants.
  # Place etminan-verifier-snapshot.sh anywhere on the verifier host and point
  # this at it (example path shown — it runs on the client/FileDaemon).
  Client Run Before Job = "/usr/local/lib/etminan/etminan-verifier-snapshot.sh"
}

JobDefs {
  Name      = "EtminanAgentDefs"
  Type      = Backup
  Level     = Full
  Client    = etminan-agent-fd             # <-- site resource
  FileSet   = "Etminan-Agent-FileSet"      # from etminan-filesets.conf
  Storage   = File                         # <-- site resource
  Messages  = Standard                     # <-- site resource
  Pool      = EtminanAgentPool
  Schedule  = "EtminanAgentSchedule"
  Priority  = 15
  # No pre-job: agent state (tls/, assigned_profile, /etc/etminan-agent) is
  # written atomically (temp+fsync+rename), so a live copy is always consistent.
}

# ── Backup Jobs ───────────────────────────────────────────────────────────────

Job {
  Use  = "EtminanVerifierDefs"
  Name = "Etminan-Verifier"
}

# Define one agent Job per monitored host (each with its own Client). This is
# the template; clone it per host, overriding Name + Client.
Job {
  Use    = "EtminanAgentDefs"
  Name   = "Etminan-Agent"
  Client = etminan-agent-fd                # <-- site resource (one per host)
}

# ── Restore Jobs ──────────────────────────────────────────────────────────────
# NOTE on verifier restore: Bacula writes baseline.db.snapshot and
# baseline.db.snapshot.audit-head back to /var/backups/etminan. To bring the
# verifier live, copy them into place (see the Verifier FileSet's restore note):
#   cp /var/backups/etminan/baseline.db.snapshot \
#      /var/lib/etminan-verifier/baseline.db
#   cp /var/backups/etminan/baseline.db.snapshot.audit-head \
#      /var/lib/etminan-verifier/baseline.db.audit-head
# Never restore the live DB directly — the FileSet never captured it. The daemon
# signing key restores in place under /var/lib/etminan-verifier; re-assert its
# 0600 perms and daemon-user ownership, then restart etminan-verifierd.

Job {
  Name     = "Etminan-Verifier-Restore"
  Type     = Restore
  Client   = etminan-verifier-fd           # <-- site resource
  FileSet  = "Etminan-Verifier-FileSet"
  Storage  = File                          # <-- site resource
  Messages = Standard                      # <-- site resource
  Pool     = EtminanVerifierPool
  Where    = /                             # files carry absolute paths; restore in place
}

Job {
  Name     = "Etminan-Agent-Restore"
  Type     = Restore
  Client   = etminan-agent-fd              # <-- site resource
  FileSet  = "Etminan-Agent-FileSet"
  Storage  = File                          # <-- site resource
  Messages = Standard                      # <-- site resource
  Pool     = EtminanAgentPool
  Where    = /
}

# ── Console ACLs (restrict restores by blast radius) ──────────────────────────
# The verifier pool holds the trust store: plaintext secrets, mTLS material, and
# the RBAC daemon's signing key. Two roles keep day-to-day agent operators away
# from it — only the DR role can restore the verifier pool, and therefore the
# daemon signing key:
#   * standard-operator — agent pool only.
#   * dr-operator       — verifier + agent pools (the whole trust store,
#                         including the daemon signing key).

Console {
  Name       = etminan-standard-operator
  Password   = "standard-console-password"    # <-- CHANGE
  CommandACL = status, messages, show, run
  JobACL     = Etminan-Agent, Etminan-Agent-Restore
  PoolACL    = EtminanAgentPool
}

Console {
  Name       = etminan-dr-operator
  Password   = "dr-console-password"          # <-- CHANGE
  CommandACL = restore, run, status, messages, show, cancel
  JobACL     = Etminan-Verifier, Etminan-Verifier-Restore, Etminan-Agent, Etminan-Agent-Restore
  PoolACL    = EtminanVerifierPool, EtminanAgentPool
}
05 — Snapshot pre-job

etminan-verifier-snapshot.sh — the WAL-safe copy

etminan-verifier-snapshot.sh

Runs on the verifier host before the FileSet is read. sqlite3 .backup makes an atomic point-in-time copy, and the audit anchor is derived from that snapshot so the DB and its anchor are always the same generation — the restored verifier never looks tampered.

etminan-verifier-snapshot.sh
#!/usr/bin/env bash
#
# EXAMPLE / REFERENCE ONLY — not installed by the etminan package. Place it on
# the verifier host and point the Job's ClientRunBeforeJob at it (see the
# example path in etminan-jobs.conf).
#
# etminan-verifier-snapshot.sh — WAL-safe snapshot of the verifier trust store,
# run as a Bacula ClientRunBeforeJob so the FileSet backs up a consistent copy
# rather than the live SQLite files. Mirrors the FDS fds.db.snapshot pattern.
#
# Why a snapshot and not a raw file copy:
#   - baseline.db is SQLite in WAL mode. A naive cp of baseline.db while the
#     verifier is mid-write (or without its -wal/-shm sidecars) yields a stale
#     or corrupt restore. `sqlite3 .backup` produces an atomic, consistent
#     point-in-time copy with no sidecars needed.
#   - baseline.db.audit-head is an EXTERNAL anchor holding the highest committed
#     (seq, entry_hash) of the hash-chained audit_log. verify_chain FAILS (and
#     reports it as tampering) if the DB is BEHIND its anchor. If we copied the
#     live DB and the live anchor at two different instants, the anchor could be
#     ahead of the DB snapshot -> restore looks tampered. We therefore DERIVE the
#     anchor from the snapshot itself, so the snapshot DB and its anchor are the
#     same generation by construction — zero race window.
set -euo pipefail

DB="${ETMINAN_BASELINE_DB:-/var/lib/etminan-verifier/baseline.db}"
SNAP_DIR="${ETMINAN_BACKUP_DIR:-/var/backups/etminan}"

install -d -m 0700 "$SNAP_DIR"

if [ ! -f "$DB" ]; then
  echo "etminan snapshot: no baseline.db at $DB — nothing to snapshot" >&2
  exit 0
fi

SNAP="$SNAP_DIR/baseline.db.snapshot"

# 1) Consistent point-in-time copy of the whole DB (commits the WAL internally).
sqlite3 "$DB" ".backup '$SNAP'"

# 2) Derive the audit anchor FROM the snapshot, so DB and anchor match exactly.
#    (Restore places this as <db>.audit-head next to the restored baseline.db.)
head_row="$(sqlite3 "$SNAP" \
  "SELECT seq || ' ' || entry_hash FROM audit_log ORDER BY seq DESC LIMIT 1;")"
if [ -n "$head_row" ]; then
  printf '%s\n' "$head_row" > "$SNAP.audit-head"
else
  # empty audit_log (fresh verifier): no anchor yet, mirror that on restore.
  rm -f "$SNAP.audit-head"
fi

chmod 0600 "$SNAP" "$SNAP.audit-head" 2>/dev/null || true
echo "etminan snapshot: wrote $SNAP (+ .audit-head) from $DB"
06 — Using a different backup tool

Not on Bacula? Reproduce these three responsibilities.

The example files are Bacula-shaped, but nothing about the requirement is. Whatever tool you run — restic, Borg, Veeam, bareos, ZFS snapshots — a correct Etminan DR backup must do all three of these. Port the intent, not the syntax. (The daemon signing key is part of the trust store here, not a separate off-host job — that is the difference from the old operator-key model.)

Snapshot the WAL DB before reading it — never copy the live file

Run the equivalent of a pre-backup hook that produces a consistent copy of baseline.db and a matching audit anchor, then back up the snapshot. In Bacula that's ClientRunBeforeJob; elsewhere it's a pre-hook (restic backup --stdin-from-command, a systemd ExecStartPre, a wrapper script). The shipped etminan-verifier-snapshot.sh is tool-agnostic — call it from any of them.

Exclude the live baseline.db, -wal, -shm and live .audit-head from the backup set, exactly as the FileSet does.

Capture the whole verifier trust-store set together

One consistent set: the snapshot + its .audit-head, state.json, tls/, and /etc/etminan-verifier (which holds plaintext integration secrets and the plugin allowlist pins — the conf pins and the pinned binaries must restore together). Restore places the snapshot back; copy it over baseline.db and its anchor as the file's restore note describes. Agent state is optional and re-derivable.

Protect the daemon signing key on backup and restore

In the daemon (op) model there are no off-host operator keys: the etminan-verifierd daemon holds one signing key (/var/lib/etminan-verifier/daemon-signing.key, 0600, daemon-owned) on the verifier host, and it is captured by the trust-store set in the step above. Because whoever can restore that set can restore the signing key, lock restore access down — encrypt the volume at rest and gate it to a DR/custodian role, exactly as the two Console ACLs do. On restore, re-assert the key's 0600 perms and daemon-user ownership, then restart etminan-verifierd.

TPM AK/EK key material is in-TPM only and cannot be backed up by any tool — a TPM clear forces re-enrollment of that host regardless.