Almost all of an Etminan deployment is re-installable from the package and re-derivable by re-enrolling. Exactly one thing is not: the verifier trust store — the baselines, the hash-chained audit log, the pinned host fingerprints, and this verifier's mTLS identity. Lose it and you re-enroll every monitored host from scratch. This page is the disaster-recovery plan for that host, plus three ready-to-adapt example files for Bacula.
These three files are a Bacula example — but the backup itself is required.
They are what a correct disaster-recovery backup of an Etminan deployment needs to capture.
The files are written for Bacula specifically (FileSets, Jobs, a
ClientRunBeforeJob snapshot).
If you use a different backup solution — restic, Borg, Veeam, Duplicati, bareos, a filesystem snapshot, anything — you must adapt them accordingly: the file formats won't apply, but the three responsibilities they encode (a WAL-safe DB snapshot, the trust-store file set — which now includes the RBAC daemon's signing key — and locking down restore access to that key) still do. None of these files is installed by the package; they are examples you copy onto your backup infrastructure.
Back up by blast radius, not by host count. The table is the whole priority order.
| Data | Where | If lost | Priority |
|---|---|---|---|
| Verifier trust store | the verifier host — baseline.db (SQLite/WAL), its audit-chain anchor,
state.json, tls/, /etc/etminan-verifier |
Catastrophic. Baselines, audit history and every pinned host fingerprint are gone; re-enroll every host and rebuild every baseline. | Must back up |
| RBAC daemon signing key | on the verifier host — /var/lib/etminan-verifier/daemon-signing.key
(0600, owned by the daemon user). Operators hold no key of their own. |
Cannot re-verify existing signed baseline batches. It lives on the verifier by design, so it is captured by the verifier trust-store backup. | Part of the trust store |
| Agent state | each monitored host — tls/, assigned_profile,
/etc/etminan-agent |
Re-derivable: re-enroll or op rotate-tls from the verifier. Backed up only to make restore a copy instead of a re-enrollment. | Optional / nice |
| TPM AK/EK key material | inside each host's TPM | Not backup-able by design. A TPM clear forces re-enrollment of that host regardless of any backup. | Cannot back up |
The verifier DB is SQLite in WAL mode. A naive cp of a live
baseline.db yields a stale or corrupt restore, and its external
audit-chain anchor can end up ahead of the copied DB — which the verifier reports as
tampering. That is the entire reason for the snapshot pre-job below, and the reason
every backup tool needs adapting rather than pointing at the raw file.
Two go on the Director; one runs on the verifier host. Each is heavily commented — the comments are the documentation. Download them here and read the full source below.
What to back up. Defines the Verifier and Agent FileSets — the verifier one includes the daemon signing key and excludes the live SQLite DB in favour of the snapshot.
Include first, before the Jobs file.
etminan-filesets.confWhen and how. Schedules, Pools, JobDefs and Backup/Restore Jobs that reference those FileSets, wire in the pre-job, and split restore access with Console ACLs.
Companion to the FileSets file.
etminan-jobs.confThe ClientRunBeforeJob that produces a WAL-safe
.backup snapshot of baseline.db
plus a matching audit-chain anchor — zero race window by construction.
Needs: sqlite3
Two FileSets: the catastrophic verifier trust store (backing up the snapshot, never the live DB, and including the RBAC daemon's signing key) and the mostly-re-derivable per-host agent state. There is no separate operator-keys set — in the daemon model operators hold no key.
# Bacula FileSets for Etminan disaster recovery.
#
# EXAMPLE / REFERENCE ONLY — this is not installed by the etminan package and is
# not part of the deliverable. Copy it onto your Bacula Director and adapt the
# paths and the "<-- site resource" names to your own deployment.
#
# A generic backup config for an Etminan deployment — Etminan has exactly two
# installable parts, so there are two installation FileSets:
# * Etminan-Verifier-FileSet — the verifier install (the trust store; the only
# catastrophic-loss host). Includes the RBAC
# daemon's signing key — see the note below.
# * Etminan-Agent-FileSet — an agent install, per monitored host; mostly
# re-derivable. Cloned per host in the Jobs file.
#
# NO separate operator-keys FileSet. In the daemon (op) model, operators hold NO
# signing key at all: the etminan-verifierd daemon holds a single signing key on
# the verifier host and signs each authorised action on the operator's behalf,
# and operator identities are UID->role/scope rows inside baseline.db. So the
# only signing key to back up lives on the verifier and is captured by the
# Verifier FileSet below — there is no off-host key store to protect separately.
#
# NOTE: nothing here is created by any install script — an Etminan deployment is
# installed from the package; the operator hand-creates the real config, TLS
# identity, and (on the verifier) the DB/state. That operator-created state is
# exactly what these FileSets capture, since the package ships only *.example
# templates and empty directories.
#
# IMPORTANT — pair the verifier FileSet with the snapshot pre-job. baseline.db
# is SQLite/WAL; this FileSet backs up the *snapshot* produced by
# etminan-verifier-snapshot.sh, never the live DB. The Jobs, Schedules, Pools,
# and the ClientRunBeforeJob wiring live in the companion etminan-jobs.conf.
# Include both from your Director config, FileSets first:
#
# @/etc/bacula/etminan/etminan-filesets.conf # example path — put them
# @/etc/bacula/etminan/etminan-jobs.conf # wherever your Director reads
FileSet {
Name = "Etminan-Verifier-FileSet"
Include {
Options {
signature = SHA256
compression = GZIP9
# Never back up the live SQLite DB or its sidecars — the snapshot below
# is the consistent copy. Excluding these avoids a corrupt/stale restore.
WildFile = "/var/lib/etminan-verifier/baseline.db"
WildFile = "/var/lib/etminan-verifier/baseline.db-wal"
WildFile = "/var/lib/etminan-verifier/baseline.db-shm"
WildFile = "/var/lib/etminan-verifier/baseline.db.audit-head"
Exclude = yes
}
Options {
signature = SHA256
compression = GZIP9
}
# --- The trust store (CATASTROPHIC if lost) ---------------------------
# WAL-safe SQLite snapshot + its consistent audit-chain anchor, produced by
# ClientRunBeforeJob = etminan-verifier-snapshot.sh. Restore: copy
# baseline.db.snapshot -> baseline.db and baseline.db.snapshot.audit-head ->
# baseline.db.audit-head (see the exclude block above — never the live DB).
# The snapshot also carries the operator identity registry (UID->role/scope)
# and all signed baseline batches, since both live inside baseline.db.
File = /var/backups/etminan/baseline.db.snapshot
File = /var/backups/etminan/baseline.db.snapshot.audit-head
# The RBAC daemon's signing key (CROWN JEWEL). etminan-verifierd holds this
# single Ed25519 seed and signs every authorised operator action with it;
# operators hold no key of their own. It lives on the verifier host by
# design, so it is part of this trust store — restore it together with the
# DB snapshot so signed batches keep verifying. 0600, owned by the daemon
# user: preserve those perms/owner on restore (see the Console ACL note in
# etminan-jobs.conf — only the DR role can restore this pool).
File = /var/lib/etminan-verifier/daemon-signing.key
# Per-host enrollment + replay state: pinned AK/EK/TLS fingerprints, PCR10
# cursor, signed enrollment/rotation accountability. Written atomically
# (temp+fsync+rename) so a live copy is always consistent — safe as-is.
# Losing it forces re-enrollment of every host from scratch.
File = /var/lib/etminan-verifier/state.json
# This verifier's mTLS identity. Its fingerprint is pinned into every agent
# (ETMINAN_VERIFIER_CERT_FINGERPRINT). Losing it forces keygen-tls + re-pin
# on every agent by hand.
File = /var/lib/etminan-verifier/tls
# --- Config + plugin allowlists (recoverable, but back up) ------------
# verifier.env holds plaintext integration secrets (RT/FreeITSM/PagerDuty/
# Slack/webhook). The two *-plugins.conf files carry the SHA256 pins that
# authorize plugin execution; the *-plugins/ dirs hold the pinned binaries.
# The conf pins and the binaries must restore together.
File = /etc/etminan-verifier
# --- Convenience: fast-restore binary (re-installable from package) ---
WildFile = "/usr/bin/etminan-verifier"
File = /usr/bin
# --- NOT captured here (out of band): -------------------------------------
# * Plugin-catalog signing key (~/.etminan-catalog-signing.key) — offline,
# never on a verifier host (HANDBOOK §5). Only exists if you self-sign
# your own private plugin catalog; most deployments consume the vendor
# catalog, whose private half never touches a customer host at all.
# * The host TPM is the irreplaceable factor for AGENTS: a TPM clear/reset
# invalidates the AK/EK pinned in state.json and forces re-enroll. TPM
# key material cannot be backed up.
}
}
FileSet {
Name = "Etminan-Agent-FileSet"
Include {
Options {
signature = SHA256
compression = GZIP9
}
# Agent mTLS identity — fingerprint is pinned in the verifier's state.json.
# Loss requires re-enroll or op rotate-tls (verifier-side action), not fatal.
File = /var/lib/etminan-agent/tls
# Last profile pushed by op assign-profile; re-derivable (authoritative record
# is in the verifier's state.json + audit_log), included for a clean restore.
File = /var/lib/etminan-agent/assigned_profile
# agent.env (holds ETMINAN_VERIFIER_CERT_FINGERPRINT, the pin) + profiles.conf
File = /etc/etminan-agent
WildFile = "/usr/bin/etminan-agent"
File = /usr/bin
# NOTE: TPM AK/EK are in-TPM only — nothing on disk to back up. A TPM clear
# forces re-enrollment regardless of this backup.
}
}
Schedules, Pools, JobDefs and Jobs. The verifier JobDef wires the snapshot in as Client Run Before Job; separate pools plus two Console ACL roles keep day-to-day agent operators away from the verifier pool — only the DR role can restore the trust store, and therefore the daemon signing key.
# Bacula Jobs for Etminan disaster recovery.
#
# EXAMPLE / REFERENCE ONLY — not installed by the etminan package, not part of
# the deliverable. Copy onto your Bacula Director and adapt to your deployment.
#
# This is the companion to etminan-filesets.conf — that file defines *what* to
# back up (Etminan-Verifier-FileSet and Etminan-Agent-FileSet); this file
# defines *when* and *how* (Schedules, Pools, JobDefs, and the Backup/Restore
# Jobs that reference those FileSets). Without this file the FileSets are never
# exercised.
#
# Include both from your Director config, FileSets first:
#
# @/etc/bacula/etminan/etminan-filesets.conf # example path — put them
# @/etc/bacula/etminan/etminan-jobs.conf # wherever your Director reads
#
# The Director, Catalog, Messages, Client (FileDaemon), and Storage resources
# are site-specific and NOT defined here — supply them from your environment.
# The names referenced below and marked "<-- site resource" must match yours:
#
# Client = etminan-verifier-fd , etminan-agent-fd
# Storage = File
# Messages = Standard
# Catalog = MyCatalog
#
# There is NO operator-workstation client. In the daemon (op) model, operators
# hold no signing key — the etminan-verifierd daemon holds the single signing
# key on the verifier host, so it is captured by the verifier backup, not a
# separate off-host job (see etminan-filesets.conf).
#
# See Bacula-Forgejo/docs/bacula-director.conf.example for a complete Director/
# Catalog/Storage/Messages scaffold to adapt.
# ── Schedules ─────────────────────────────────────────────────────────────────
# Weekly Full + daily Incremental. The verifier snapshot pre-job regenerates
# baseline.db.snapshot on every run, so Incrementals reliably capture DB changes.
Schedule {
Name = "EtminanVerifierSchedule"
Run = Full 1st sun at 01:00
Run = Incremental mon-sat at 01:00
}
Schedule {
Name = "EtminanAgentSchedule"
# Agent state is small and mostly re-derivable — a nightly Full is cheap and
# keeps restore dead-simple (no chain to replay).
Run = Full daily at 02:00
}
# ── Pools ─────────────────────────────────────────────────────────────────────
# The verifier trust store is the only catastrophic-loss data AND contains
# plaintext secrets (verifier.env), this verifier's mTLS material, and the RBAC
# daemon's signing key. It gets its own restricted pool so restore access can be
# locked down via Console ACLs below — whoever can restore this pool can restore
# the daemon signing key, so keep that role tight.
Pool {
Name = EtminanVerifierPool
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 365 days
Maximum Volume Bytes = 20G
Label Format = "EtminanVerifierVol-"
}
Pool {
Name = EtminanAgentPool
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 180 days
Maximum Volume Bytes = 20G
Label Format = "EtminanAgentVol-"
}
# ── JobDefs ───────────────────────────────────────────────────────────────────
JobDefs {
Name = "EtminanVerifierDefs"
Type = Backup
Level = Incremental
Client = etminan-verifier-fd # <-- site resource
FileSet = "Etminan-Verifier-FileSet" # from etminan-filesets.conf
Storage = File # <-- site resource
Messages = Standard # <-- site resource
Pool = EtminanVerifierPool
Schedule = "EtminanVerifierSchedule"
Priority = 10
# CRITICAL: take the WAL-safe snapshot BEFORE the FileSet is read, so the
# backup captures baseline.db.snapshot (+ .audit-head) instead of the live
# SQLite DB. The FileSet excludes the live DB; this produces the copy it wants.
# Place etminan-verifier-snapshot.sh anywhere on the verifier host and point
# this at it (example path shown — it runs on the client/FileDaemon).
Client Run Before Job = "/usr/local/lib/etminan/etminan-verifier-snapshot.sh"
}
JobDefs {
Name = "EtminanAgentDefs"
Type = Backup
Level = Full
Client = etminan-agent-fd # <-- site resource
FileSet = "Etminan-Agent-FileSet" # from etminan-filesets.conf
Storage = File # <-- site resource
Messages = Standard # <-- site resource
Pool = EtminanAgentPool
Schedule = "EtminanAgentSchedule"
Priority = 15
# No pre-job: agent state (tls/, assigned_profile, /etc/etminan-agent) is
# written atomically (temp+fsync+rename), so a live copy is always consistent.
}
# ── Backup Jobs ───────────────────────────────────────────────────────────────
Job {
Use = "EtminanVerifierDefs"
Name = "Etminan-Verifier"
}
# Define one agent Job per monitored host (each with its own Client). This is
# the template; clone it per host, overriding Name + Client.
Job {
Use = "EtminanAgentDefs"
Name = "Etminan-Agent"
Client = etminan-agent-fd # <-- site resource (one per host)
}
# ── Restore Jobs ──────────────────────────────────────────────────────────────
# NOTE on verifier restore: Bacula writes baseline.db.snapshot and
# baseline.db.snapshot.audit-head back to /var/backups/etminan. To bring the
# verifier live, copy them into place (see the Verifier FileSet's restore note):
# cp /var/backups/etminan/baseline.db.snapshot \
# /var/lib/etminan-verifier/baseline.db
# cp /var/backups/etminan/baseline.db.snapshot.audit-head \
# /var/lib/etminan-verifier/baseline.db.audit-head
# Never restore the live DB directly — the FileSet never captured it. The daemon
# signing key restores in place under /var/lib/etminan-verifier; re-assert its
# 0600 perms and daemon-user ownership, then restart etminan-verifierd.
Job {
Name = "Etminan-Verifier-Restore"
Type = Restore
Client = etminan-verifier-fd # <-- site resource
FileSet = "Etminan-Verifier-FileSet"
Storage = File # <-- site resource
Messages = Standard # <-- site resource
Pool = EtminanVerifierPool
Where = / # files carry absolute paths; restore in place
}
Job {
Name = "Etminan-Agent-Restore"
Type = Restore
Client = etminan-agent-fd # <-- site resource
FileSet = "Etminan-Agent-FileSet"
Storage = File # <-- site resource
Messages = Standard # <-- site resource
Pool = EtminanAgentPool
Where = /
}
# ── Console ACLs (restrict restores by blast radius) ──────────────────────────
# The verifier pool holds the trust store: plaintext secrets, mTLS material, and
# the RBAC daemon's signing key. Two roles keep day-to-day agent operators away
# from it — only the DR role can restore the verifier pool, and therefore the
# daemon signing key:
# * standard-operator — agent pool only.
# * dr-operator — verifier + agent pools (the whole trust store,
# including the daemon signing key).
Console {
Name = etminan-standard-operator
Password = "standard-console-password" # <-- CHANGE
CommandACL = status, messages, show, run
JobACL = Etminan-Agent, Etminan-Agent-Restore
PoolACL = EtminanAgentPool
}
Console {
Name = etminan-dr-operator
Password = "dr-console-password" # <-- CHANGE
CommandACL = restore, run, status, messages, show, cancel
JobACL = Etminan-Verifier, Etminan-Verifier-Restore, Etminan-Agent, Etminan-Agent-Restore
PoolACL = EtminanVerifierPool, EtminanAgentPool
}
Runs on the verifier host before the FileSet is read. sqlite3 .backup makes an atomic point-in-time copy, and the audit anchor is derived from that snapshot so the DB and its anchor are always the same generation — the restored verifier never looks tampered.
#!/usr/bin/env bash
#
# EXAMPLE / REFERENCE ONLY — not installed by the etminan package. Place it on
# the verifier host and point the Job's ClientRunBeforeJob at it (see the
# example path in etminan-jobs.conf).
#
# etminan-verifier-snapshot.sh — WAL-safe snapshot of the verifier trust store,
# run as a Bacula ClientRunBeforeJob so the FileSet backs up a consistent copy
# rather than the live SQLite files. Mirrors the FDS fds.db.snapshot pattern.
#
# Why a snapshot and not a raw file copy:
# - baseline.db is SQLite in WAL mode. A naive cp of baseline.db while the
# verifier is mid-write (or without its -wal/-shm sidecars) yields a stale
# or corrupt restore. `sqlite3 .backup` produces an atomic, consistent
# point-in-time copy with no sidecars needed.
# - baseline.db.audit-head is an EXTERNAL anchor holding the highest committed
# (seq, entry_hash) of the hash-chained audit_log. verify_chain FAILS (and
# reports it as tampering) if the DB is BEHIND its anchor. If we copied the
# live DB and the live anchor at two different instants, the anchor could be
# ahead of the DB snapshot -> restore looks tampered. We therefore DERIVE the
# anchor from the snapshot itself, so the snapshot DB and its anchor are the
# same generation by construction — zero race window.
set -euo pipefail
DB="${ETMINAN_BASELINE_DB:-/var/lib/etminan-verifier/baseline.db}"
SNAP_DIR="${ETMINAN_BACKUP_DIR:-/var/backups/etminan}"
install -d -m 0700 "$SNAP_DIR"
if [ ! -f "$DB" ]; then
echo "etminan snapshot: no baseline.db at $DB — nothing to snapshot" >&2
exit 0
fi
SNAP="$SNAP_DIR/baseline.db.snapshot"
# 1) Consistent point-in-time copy of the whole DB (commits the WAL internally).
sqlite3 "$DB" ".backup '$SNAP'"
# 2) Derive the audit anchor FROM the snapshot, so DB and anchor match exactly.
# (Restore places this as <db>.audit-head next to the restored baseline.db.)
head_row="$(sqlite3 "$SNAP" \
"SELECT seq || ' ' || entry_hash FROM audit_log ORDER BY seq DESC LIMIT 1;")"
if [ -n "$head_row" ]; then
printf '%s\n' "$head_row" > "$SNAP.audit-head"
else
# empty audit_log (fresh verifier): no anchor yet, mirror that on restore.
rm -f "$SNAP.audit-head"
fi
chmod 0600 "$SNAP" "$SNAP.audit-head" 2>/dev/null || true
echo "etminan snapshot: wrote $SNAP (+ .audit-head) from $DB"
The example files are Bacula-shaped, but nothing about the requirement is. Whatever tool you run — restic, Borg, Veeam, bareos, ZFS snapshots — a correct Etminan DR backup must do all three of these. Port the intent, not the syntax. (The daemon signing key is part of the trust store here, not a separate off-host job — that is the difference from the old operator-key model.)
Run the equivalent of a pre-backup hook that produces a consistent copy of
baseline.db and a matching audit anchor, then back up the
snapshot. In Bacula that's ClientRunBeforeJob; elsewhere
it's a pre-hook (restic backup --stdin-from-command, a
systemd ExecStartPre, a wrapper script). The shipped
etminan-verifier-snapshot.sh is tool-agnostic — call it from
any of them.
Exclude the live baseline.db,
-wal, -shm and live
.audit-head from the backup set, exactly as the FileSet does.
One consistent set: the snapshot + its .audit-head,
state.json, tls/, and
/etc/etminan-verifier (which holds plaintext integration
secrets and the plugin allowlist pins — the conf pins and the pinned binaries must
restore together). Restore places the snapshot back; copy it over
baseline.db and its anchor as the file's restore note
describes. Agent state is optional and re-derivable.
In the daemon (op) model there are no off-host operator keys: the
etminan-verifierd daemon holds one signing key
(/var/lib/etminan-verifier/daemon-signing.key, 0600, daemon-owned)
on the verifier host, and it is captured by the trust-store set in the step above. Because
whoever can restore that set can restore the signing key, lock restore access down —
encrypt the volume at rest and gate it to a DR/custodian role, exactly as the two Console
ACLs do. On restore, re-assert the key's 0600 perms and
daemon-user ownership, then restart etminan-verifierd.
TPM AK/EK key material is in-TPM only and cannot be backed up by any tool — a TPM clear forces re-enrollment of that host regardless.