Almost all of an Etminan deployment is re-installable from the package and re-derivable by re-enrolling. Exactly one thing is not: the verifier trust store — the baselines, the hash-chained audit log, the pinned host fingerprints, and this verifier's mTLS identity. Lose it and you re-enroll every monitored host from scratch. This page is the disaster-recovery plan for that host, plus three ready-to-adapt example files for Bacula.
These three files are a Bacula example — but the backup itself is required.
They are what a correct disaster-recovery backup of an Etminan deployment needs to capture.
The files are written for Bacula specifically (FileSets, Jobs, a
ClientRunBeforeJob snapshot).
If you use a different backup solution — restic, Borg, Veeam, Duplicati, bareos, a filesystem snapshot, anything — you must adapt them accordingly: the file formats won't apply, but the three responsibilities they encode (a WAL-safe DB snapshot, the trust-store file set, and off-host operator keys) still do. None of these files is installed by the package; they are examples you copy onto your backup infrastructure.
Back up by blast radius, not by host count. The table is the whole priority order.
| Data | Where | If lost | Priority |
|---|---|---|---|
| Verifier trust store | the verifier host — baseline.db (SQLite/WAL), its audit-chain anchor,
state.json, tls/, /etc/etminan-verifier |
Catastrophic. Baselines, audit history and every pinned host fingerprint are gone; re-enroll every host and rebuild every baseline. | Must back up |
| Operator signing keys | off-host — an operator workstation, never a verifier. No default path
(keygen --out <path>). |
Cannot re-sign existing baseline batches. Kept deliberately separate so a compromised verifier never also yields the keys that authorize its baselines. | Back up separately |
| Agent state | each monitored host — tls/, assigned_profile,
/etc/etminan-agent |
Re-derivable: re-enroll or rotate-tls from the verifier. Backed up only to make restore a copy instead of a re-enrollment. | Optional / nice |
| TPM AK/EK key material | inside each host's TPM | Not backup-able by design. A TPM clear forces re-enrollment of that host regardless of any backup. | Cannot back up |
The verifier DB is SQLite in WAL mode. A naive cp of a live
baseline.db yields a stale or corrupt restore, and its external
audit-chain anchor can end up ahead of the copied DB — which the verifier reports as
tampering. That is the entire reason for the snapshot pre-job below, and the reason
every backup tool needs adapting rather than pointing at the raw file.
Two go on the Director; one runs on the verifier host. Each is heavily commented — the comments are the documentation. Download them here and read the full source below.
What to back up. Defines the Verifier, Agent, and optional off-host Operator-Keys FileSets — including excluding the live SQLite DB in favour of the snapshot.
Include first, before the Jobs file.
etminan-filesets.confWhen and how. Schedules, Pools, JobDefs and Backup/Restore Jobs that reference those FileSets, wire in the pre-job, and split restore access with Console ACLs.
Companion to the FileSets file.
etminan-jobs.confThe ClientRunBeforeJob that produces a WAL-safe
.backup snapshot of baseline.db
plus a matching audit-chain anchor — zero race window by construction.
Needs: sqlite3
Three FileSets: the catastrophic verifier trust store (backing up the snapshot, never the live DB), the mostly-re-derivable per-host agent state, and an optional, security-sensitive off-host operator-keys set. Read the inline security note before enabling the last one.
# Bacula FileSets for Etminan disaster recovery.
#
# EXAMPLE / REFERENCE ONLY — this is not installed by the etminan package and is
# not part of the deliverable. Copy it onto your Bacula Director and adapt the
# paths and the "<-- site resource" names to your own deployment.
#
# A generic backup config for an Etminan deployment — Etminan has exactly two
# installable parts, so there are two installation FileSets:
# * Etminan-Verifier-FileSet — the verifier install (the trust store; the only
# catastrophic-loss host).
# * Etminan-Agent-FileSet — an agent install, per monitored host; mostly
# re-derivable. Cloned per host in the Jobs file.
#
# Plus one OPTIONAL, off-host FileSet that is not an installation but is a
# critical operator secret:
# * Etminan-Operator-Keys-FileSet — the operator's baseline-approval signing
# key(s), held on an operator workstation.
# Security-sensitive: read the warning above
# that FileSet before enabling it.
#
# NOTE: nothing here is created by any install script — an Etminan deployment is
# installed from the package; the operator hand-creates the real config, TLS
# identity, and (on the verifier) the DB/state. That operator-created state is
# exactly what these FileSets capture, since the package ships only *.example
# templates and empty directories.
#
# IMPORTANT — pair the verifier FileSet with the snapshot pre-job. baseline.db
# is SQLite/WAL; this FileSet backs up the *snapshot* produced by
# etminan-verifier-snapshot.sh, never the live DB. The Jobs, Schedules, Pools,
# and the ClientRunBeforeJob wiring live in the companion etminan-jobs.conf.
# Include both from your Director config, FileSets first:
#
# @/etc/bacula/etminan/etminan-filesets.conf # example path — put them
# @/etc/bacula/etminan/etminan-jobs.conf # wherever your Director reads
FileSet {
Name = "Etminan-Verifier-FileSet"
Include {
Options {
signature = SHA256
compression = GZIP9
# Never back up the live SQLite DB or its sidecars — the snapshot below
# is the consistent copy. Excluding these avoids a corrupt/stale restore.
WildFile = "/var/lib/etminan-verifier/baseline.db"
WildFile = "/var/lib/etminan-verifier/baseline.db-wal"
WildFile = "/var/lib/etminan-verifier/baseline.db-shm"
WildFile = "/var/lib/etminan-verifier/baseline.db.audit-head"
Exclude = yes
}
Options {
signature = SHA256
compression = GZIP9
}
# --- The trust store (CATASTROPHIC if lost) ---------------------------
# WAL-safe SQLite snapshot + its consistent audit-chain anchor, produced by
# ClientRunBeforeJob = etminan-verifier-snapshot.sh. Restore: copy
# baseline.db.snapshot -> baseline.db and baseline.db.snapshot.audit-head ->
# baseline.db.audit-head (see the exclude block above — never the live DB).
File = /var/backups/etminan/baseline.db.snapshot
File = /var/backups/etminan/baseline.db.snapshot.audit-head
# Per-host enrollment + replay state: pinned AK/EK/TLS fingerprints, PCR10
# cursor, signed enrollment/rotation accountability. Written atomically
# (temp+fsync+rename) so a live copy is always consistent — safe as-is.
# Losing it forces re-enrollment of every host from scratch.
File = /var/lib/etminan-verifier/state.json
# This verifier's mTLS identity. Its fingerprint is pinned into every agent
# (ETMINAN_VERIFIER_CERT_FINGERPRINT). Losing it forces keygen-tls + re-pin
# on every agent by hand.
File = /var/lib/etminan-verifier/tls
# --- Config + plugin allowlists (recoverable, but back up) ------------
# verifier.env holds plaintext integration secrets (RT/FreeITSM/PagerDuty/
# Slack/webhook). The two *-plugins.conf files carry the SHA256 pins that
# authorize plugin execution; the *-plugins/ dirs hold the pinned binaries.
# The conf pins and the binaries must restore together.
File = /etc/etminan-verifier
# --- Convenience: fast-restore binary (re-installable from package) ---
WildFile = "/usr/bin/etminan-verifier"
File = /usr/bin
# --- NOT captured here (off-host / out of band): --------------------------
# * Operator baseline-approval signing keys — no default path; created by
# `keygen --out <path>`, held per-operator OFF the verifier host. Their
# private halves sign approve/reject/enroll; a lost key cannot re-sign
# existing baseline_batches. Back up per operator (their responsibility).
# * Plugin-catalog signing key (~/.etminan-catalog-signing.key) — offline,
# never on a verifier host (HANDBOOK §5).
# * The host TPM is the irreplaceable factor for AGENTS: a TPM clear/reset
# invalidates the AK/EK pinned in state.json and forces re-enroll. TPM
# key material cannot be backed up.
}
}
FileSet {
Name = "Etminan-Agent-FileSet"
Include {
Options {
signature = SHA256
compression = GZIP9
}
# Agent mTLS identity — fingerprint is pinned in the verifier's state.json.
# Loss requires re-enroll or rotate-tls (verifier-side action), not fatal.
File = /var/lib/etminan-agent/tls
# Last profile pushed by assign-profile; re-derivable (authoritative record
# is in the verifier's state.json + audit_log), included for a clean restore.
File = /var/lib/etminan-agent/assigned_profile
# agent.env (holds ETMINAN_VERIFIER_CERT_FINGERPRINT, the pin) + profiles.conf
File = /etc/etminan-agent
WildFile = "/usr/bin/etminan-agent"
File = /usr/bin
# NOTE: TPM AK/EK are in-TPM only — nothing on disk to back up. A TPM clear
# forces re-enrollment regardless of this backup.
}
}
# ── Etminan-Operator-Keys-FileSet — OFF-HOST, read this before enabling ───────
# The crown-jewel private keys that are deliberately NOT stored on any verifier
# or agent (that separation is the whole point — a compromised verifier host must
# not also yield the keys that authorize its baselines):
#
# * Operator baseline-approval signing key(s) — sign enroll/rotate/approve/
# reject/assign-profile. No default path; created per-operator with
# `etminan-verifier keygen --out <path>` (writes <path> + <path>.pub). A
# lost private half cannot re-sign existing baseline_batches. This is the
# key a customer deployment actually has.
# * (Only if you run your OWN private plugin catalog) a catalog signing key
# from `etminan-verifier plugins keygen-catalog`. Most deployments don't —
# they consume the vendor catalog, whose public half is compiled in and
# whose private half never exists on a customer host at all. Skip unless
# you self-sign a catalog.
#
# SECURITY — do NOT undo the separation this design relies on:
# * Back these up ONLY from the operator workstation that legitimately holds
# them, into their OWN pool (EtminanOperatorKeysPool in etminan-jobs.conf) —
# NEVER the verifier pool. A single pool restore must never hand someone both
# the baselines AND the keys that authorize them.
# * Prefer an encrypted, offline medium. If you do use Bacula, enable client-
# side Data Encryption (PKI Encryption/PKI Keypair on the operator
# workstation's FileDaemon) so the keys are encrypted at rest in the volume.
# * This FileSet is the Bacula option for shops that already back up operator
# workstations — not a recommendation to put signing keys on the same tape
# as everything else.
FileSet {
Name = "Etminan-Operator-Keys-FileSet"
Include {
Options {
signature = SHA256
compression = GZIP9
# See the SECURITY note above — configure PKI Encryption on this client's
# FileDaemon so these private keys are encrypted at rest in the volume.
}
# Fill in the ACTUAL paths on this operator's workstation. Operator signing
# keys have no default location (keygen --out <path>) — these are examples:
File = /home/<operator>/.etminan/operator-signing.key
File = /home/<operator>/.etminan/operator-signing.key.pub
# Only if you self-sign your own plugin catalog (uncommon — see above):
# File = /home/<operator>/.etminan-catalog-signing.key
}
}
Schedules, Pools, JobDefs and Jobs. The verifier JobDef wires the snapshot in as Client Run Before Job; separate pools plus three Console ACL roles keep any single console from restoring both the baselines and the keys that authorize them.
# Bacula Jobs for Etminan disaster recovery.
#
# EXAMPLE / REFERENCE ONLY — not installed by the etminan package, not part of
# the deliverable. Copy onto your Bacula Director and adapt to your deployment.
#
# This is the companion to etminan-filesets.conf — that file defines *what* to
# back up (Etminan-Verifier-FileSet, Etminan-Agent-FileSet, and the optional
# Etminan-Operator-Keys-FileSet); this file defines *when* and *how* (Schedules,
# Pools, JobDefs, and the Backup/Restore Jobs that reference those FileSets).
# Without this file the FileSets are never exercised.
#
# Include both from your Director config, FileSets first:
#
# @/etc/bacula/etminan/etminan-filesets.conf # example path — put them
# @/etc/bacula/etminan/etminan-jobs.conf # wherever your Director reads
#
# The Director, Catalog, Messages, Client (FileDaemon), and Storage resources
# are site-specific and NOT defined here — supply them from your environment.
# The names referenced below and marked "<-- site resource" must match yours:
#
# Client = etminan-verifier-fd , etminan-agent-fd , etminan-operator-fd
# Storage = File
# Messages = Standard
# Catalog = MyCatalog
#
# etminan-operator-fd is the OPERATOR WORKSTATION that holds the signing keys —
# deliberately a different client from any verifier (see the security note on
# Etminan-Operator-Keys-FileSet in etminan-filesets.conf).
#
# See Bacula-Forgejo/docs/bacula-director.conf.example for a complete Director/
# Catalog/Storage/Messages scaffold to adapt.
# ── Schedules ─────────────────────────────────────────────────────────────────
# Weekly Full + daily Incremental. The verifier snapshot pre-job regenerates
# baseline.db.snapshot on every run, so Incrementals reliably capture DB changes.
Schedule {
Name = "EtminanVerifierSchedule"
Run = Full 1st sun at 01:00
Run = Incremental mon-sat at 01:00
}
Schedule {
Name = "EtminanAgentSchedule"
# Agent state is small and mostly re-derivable — a nightly Full is cheap and
# keeps restore dead-simple (no chain to replay).
Run = Full daily at 02:00
}
Schedule {
Name = "EtminanOperatorKeysSchedule"
# Signing keys change rarely (only on rotation). A nightly Full is trivially
# cheap and means a fresh key is captured within a day of being generated.
Run = Full daily at 04:00
}
# ── Pools ─────────────────────────────────────────────────────────────────────
# The verifier trust store is the only catastrophic-loss data AND contains
# plaintext secrets (verifier.env) plus mTLS/signing material. It gets its own
# restricted pool so restore access can be locked down via Console ACLs below.
Pool {
Name = EtminanVerifierPool
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 365 days
Maximum Volume Bytes = 20G
Label Format = "EtminanVerifierVol-"
}
Pool {
Name = EtminanAgentPool
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 180 days
Maximum Volume Bytes = 20G
Label Format = "EtminanAgentVol-"
}
# The operator-keys pool holds the private signing keys and is kept in its OWN
# pool so restore can be locked to a key custodian (see Console ACLs below) and
# NEVER mixed with the verifier trust store. Long retention: these keys are
# long-lived, so keep history for the working life of the key, not just weeks.
Pool {
Name = EtminanOperatorKeysPool
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 3650 days
Maximum Volume Bytes = 2G
Label Format = "EtminanOperatorKeysVol-"
}
# ── JobDefs ───────────────────────────────────────────────────────────────────
JobDefs {
Name = "EtminanVerifierDefs"
Type = Backup
Level = Incremental
Client = etminan-verifier-fd # <-- site resource
FileSet = "Etminan-Verifier-FileSet" # from etminan-filesets.conf
Storage = File # <-- site resource
Messages = Standard # <-- site resource
Pool = EtminanVerifierPool
Schedule = "EtminanVerifierSchedule"
Priority = 10
# CRITICAL: take the WAL-safe snapshot BEFORE the FileSet is read, so the
# backup captures baseline.db.snapshot (+ .audit-head) instead of the live
# SQLite DB. The FileSet excludes the live DB; this produces the copy it wants.
# Place etminan-verifier-snapshot.sh anywhere on the verifier host and point
# this at it (example path shown — it runs on the client/FileDaemon).
Client Run Before Job = "/usr/local/lib/etminan/etminan-verifier-snapshot.sh"
}
JobDefs {
Name = "EtminanAgentDefs"
Type = Backup
Level = Full
Client = etminan-agent-fd # <-- site resource
FileSet = "Etminan-Agent-FileSet" # from etminan-filesets.conf
Storage = File # <-- site resource
Messages = Standard # <-- site resource
Pool = EtminanAgentPool
Schedule = "EtminanAgentSchedule"
Priority = 15
# No pre-job: agent state (tls/, assigned_profile, /etc/etminan-agent) is
# written atomically (temp+fsync+rename), so a live copy is always consistent.
}
JobDefs {
Name = "EtminanOperatorKeysDefs"
Type = Backup
Level = Full
Client = etminan-operator-fd # <-- OPERATOR WORKSTATION, never a verifier
FileSet = "Etminan-Operator-Keys-FileSet" # from etminan-filesets.conf
Storage = File # <-- site resource
Messages = Standard # <-- site resource
Pool = EtminanOperatorKeysPool
Schedule = "EtminanOperatorKeysSchedule"
Priority = 25
# No pre-job: signing keys are static files written 0600 O_EXCL at keygen.
}
# ── Backup Jobs ───────────────────────────────────────────────────────────────
Job {
Use = "EtminanVerifierDefs"
Name = "Etminan-Verifier"
}
# Define one agent Job per monitored host (each with its own Client). This is
# the template; clone it per host, overriding Name + Client.
Job {
Use = "EtminanAgentDefs"
Name = "Etminan-Agent"
Client = etminan-agent-fd # <-- site resource (one per host)
}
# Optional — enable only if you back up the operator workstation that holds the
# signing keys. Read the security note on Etminan-Operator-Keys-FileSet first.
Job {
Use = "EtminanOperatorKeysDefs"
Name = "Etminan-Operator-Keys"
}
# ── Restore Jobs ──────────────────────────────────────────────────────────────
# NOTE on verifier restore: Bacula writes baseline.db.snapshot and
# baseline.db.snapshot.audit-head back to /var/backups/etminan. To bring the
# verifier live, copy them into place (see the Verifier FileSet's restore note):
# cp /var/backups/etminan/baseline.db.snapshot \
# /var/lib/etminan-verifier/baseline.db
# cp /var/backups/etminan/baseline.db.snapshot.audit-head \
# /var/lib/etminan-verifier/baseline.db.audit-head
# Never restore the live DB directly — the FileSet never captured it.
Job {
Name = "Etminan-Verifier-Restore"
Type = Restore
Client = etminan-verifier-fd # <-- site resource
FileSet = "Etminan-Verifier-FileSet"
Storage = File # <-- site resource
Messages = Standard # <-- site resource
Pool = EtminanVerifierPool
Where = / # files carry absolute paths; restore in place
}
Job {
Name = "Etminan-Agent-Restore"
Type = Restore
Client = etminan-agent-fd # <-- site resource
FileSet = "Etminan-Agent-FileSet"
Storage = File # <-- site resource
Messages = Standard # <-- site resource
Pool = EtminanAgentPool
Where = /
}
# Restores the signing keys to a STAGING dir, not in place — so a restore can't
# silently clobber a live key. The custodian inspects them, then moves each into
# its real location by hand (keys are 0600; preserve that on the move).
Job {
Name = "Etminan-Operator-Keys-Restore"
Type = Restore
Client = etminan-operator-fd # <-- OPERATOR WORKSTATION
FileSet = "Etminan-Operator-Keys-FileSet"
Storage = File # <-- site resource
Messages = Standard # <-- site resource
Pool = EtminanOperatorKeysPool
Where = /var/tmp/etminan-key-restore # staging; move into place by hand
}
# ── Console ACLs (restrict restores by blast radius) ──────────────────────────
# The verifier pool holds secrets + trust-store material; the operator-keys pool
# holds the signing keys. These are split across three roles so no single console
# can restore both the baselines and the keys that authorize them:
# * standard-operator — agent pool only.
# * dr-operator — verifier + agent pools (the trust store), but NOT the
# operator-keys pool.
# * key-custodian — operator-keys pool ONLY. Deliberately cannot touch the
# verifier pool, preserving the separation of duties.
Console {
Name = etminan-standard-operator
Password = "standard-console-password" # <-- CHANGE
CommandACL = status, messages, show, run
JobACL = Etminan-Agent, Etminan-Agent-Restore
PoolACL = EtminanAgentPool
}
Console {
Name = etminan-dr-operator
Password = "dr-console-password" # <-- CHANGE
CommandACL = restore, run, status, messages, show, cancel
JobACL = Etminan-Verifier, Etminan-Verifier-Restore, Etminan-Agent, Etminan-Agent-Restore
PoolACL = EtminanVerifierPool, EtminanAgentPool
}
Console {
Name = etminan-key-custodian
Password = "key-custodian-console-password" # <-- CHANGE
CommandACL = restore, run, status, messages, show, cancel
JobACL = Etminan-Operator-Keys, Etminan-Operator-Keys-Restore
PoolACL = EtminanOperatorKeysPool
}
Runs on the verifier host before the FileSet is read. sqlite3 .backup makes an atomic point-in-time copy, and the audit anchor is derived from that snapshot so the DB and its anchor are always the same generation — the restored verifier never looks tampered.
#!/usr/bin/env bash
#
# EXAMPLE / REFERENCE ONLY — not installed by the etminan package. Place it on
# the verifier host and point the Job's ClientRunBeforeJob at it (see the
# example path in etminan-jobs.conf).
#
# etminan-verifier-snapshot.sh — WAL-safe snapshot of the verifier trust store,
# run as a Bacula ClientRunBeforeJob so the FileSet backs up a consistent copy
# rather than the live SQLite files. Mirrors the FDS fds.db.snapshot pattern.
#
# Why a snapshot and not a raw file copy:
# - baseline.db is SQLite in WAL mode. A naive cp of baseline.db while the
# verifier is mid-write (or without its -wal/-shm sidecars) yields a stale
# or corrupt restore. `sqlite3 .backup` produces an atomic, consistent
# point-in-time copy with no sidecars needed.
# - baseline.db.audit-head is an EXTERNAL anchor holding the highest committed
# (seq, entry_hash) of the hash-chained audit_log. verify_chain FAILS (and
# reports it as tampering) if the DB is BEHIND its anchor. If we copied the
# live DB and the live anchor at two different instants, the anchor could be
# ahead of the DB snapshot -> restore looks tampered. We therefore DERIVE the
# anchor from the snapshot itself, so the snapshot DB and its anchor are the
# same generation by construction — zero race window.
set -euo pipefail
DB="${ETMINAN_BASELINE_DB:-/var/lib/etminan-verifier/baseline.db}"
SNAP_DIR="${ETMINAN_BACKUP_DIR:-/var/backups/etminan}"
install -d -m 0700 "$SNAP_DIR"
if [ ! -f "$DB" ]; then
echo "etminan snapshot: no baseline.db at $DB — nothing to snapshot" >&2
exit 0
fi
SNAP="$SNAP_DIR/baseline.db.snapshot"
# 1) Consistent point-in-time copy of the whole DB (commits the WAL internally).
sqlite3 "$DB" ".backup '$SNAP'"
# 2) Derive the audit anchor FROM the snapshot, so DB and anchor match exactly.
# (Restore places this as <db>.audit-head next to the restored baseline.db.)
head_row="$(sqlite3 "$SNAP" \
"SELECT seq || ' ' || entry_hash FROM audit_log ORDER BY seq DESC LIMIT 1;")"
if [ -n "$head_row" ]; then
printf '%s\n' "$head_row" > "$SNAP.audit-head"
else
# empty audit_log (fresh verifier): no anchor yet, mirror that on restore.
rm -f "$SNAP.audit-head"
fi
chmod 0600 "$SNAP" "$SNAP.audit-head" 2>/dev/null || true
echo "etminan snapshot: wrote $SNAP (+ .audit-head) from $DB"
The example files are Bacula-shaped, but nothing about the requirement is. Whatever tool you run — restic, Borg, Veeam, bareos, ZFS snapshots — a correct Etminan DR backup must do all three of these. Port the intent, not the syntax.
Run the equivalent of a pre-backup hook that produces a consistent copy of
baseline.db and a matching audit anchor, then back up the
snapshot. In Bacula that's ClientRunBeforeJob; elsewhere
it's a pre-hook (restic backup --stdin-from-command, a
systemd ExecStartPre, a wrapper script). The shipped
etminan-verifier-snapshot.sh is tool-agnostic — call it from
any of them.
Exclude the live baseline.db,
-wal, -shm and live
.audit-head from the backup set, exactly as the FileSet does.
One consistent set: the snapshot + its .audit-head,
state.json, tls/, and
/etc/etminan-verifier (which holds plaintext integration
secrets and the plugin allowlist pins — the conf pins and the pinned binaries must
restore together). Restore places the snapshot back; copy it over
baseline.db and its anchor as the file's restore note
describes. Agent state is optional and re-derivable.
The baseline-approval signing keys deliberately live on an operator workstation, never a
verifier — so a compromised verifier can't also yield the keys that authorize its baselines.
Preserve that separation in your backup too: a separate repository / pool / tape with its
own access control, encrypted at rest, never mixed with the verifier data. The
Etminan-Operator-Keys-FileSet shows the Bacula shape; the rule
is tool-independent.
TPM AK/EK key material is in-TPM only and cannot be backed up by any tool — a TPM clear forces re-enrollment of that host regardless.