OpenBao

The day the homelab needs more than per-host SOPS, this is what catches it. It is what an always-on agent talks to instead of a human at a keyboard.

​

Its job in the four-tier model

OpenBao is T2 in the four-tier secrets model: the self-hosted source of truth for live machine credentials, and the primary AI/machine interface. An always-on agent authenticates to it as itself and gets scoped, expiring credentials with zero interactive prompts. The macOS keychain can never offer that to a headless process. It does three jobs:

• Dynamic credential issuer. AppRole (and, for the most sensitive services, TLS cert) auth methods issue short-lived, per-service credentials from a KV v2 engine and dynamic secrets engines.
• SSH certificate authority. It signs short-TTL SSH certificates for automation principals (ai-agent, ansible, ci), a hybrid model where machines carry expiring certs and humans keep their static keys. No long-lived automation key sits on disk waiting to be stolen.
• Terrakube dynamic credentials. Each workspace authenticates to a scoped native path and receives only the short-lived provider credentials it needs. Terrakube owns per-workspace run locking; OpenBao does not duplicate it.

It is the MPL-licensed fork of HashiCorp Vault 1.14.x, the last release before HashiCorp’s BSL transition in 1.15.

​

Independent service secrets engines

OpenBao hosts two separate logical engines inside one shared OpenBao security boundary. They are not one combined service. AWS engine, two layers. A dedicated broker IAM identity holds one permission only: sts:AssumeRole into broker-managed roles. Its own base key is seeded into OpenBao’s AWS engine root config once at bootstrap, write-once (a routine converge never overwrites it). Rotating it is a deliberate, separate operator action: minting a fresh key on an independent administrator path and re-seeding, never OpenBao’s own rotate-root call. This way, no long-lived AWS key sits anywhere routine automation can reach it. Behind it sits one purpose role per consumer. There’s a tf-<project> role per OpenTofu root, one terrakube-<workspace> role per Terrakube workspace, and a broader administrator-purpose role for AI and general IaC use. That broader role carries an AWS permissions boundary. It blocks privilege self-escalation, credential/identity persistence (no new IAM users, keys, or login profiles), and audit/org/billing tampering. It is not AdministratorAccess. Local and AI-agent callers reach these roles through a credential_process AWS profile backed by an AppRole whose role_id/secret_id the runtime secret injector supplies (below). Terrakube workspaces reach their own role directly through their workspace JWT identity, never through a local profile. Each engine has its own catalog entry, mount, service configuration, OpenBao policy, consumer endpoints, and leases. Normal API access to one engine does not grant access to the other. Both engines still share the OpenBao root and administrator control plane, Raft storage boundary, hosts, plugin runtime, signature-verified release artifacts, and outbound HTTPS. A compromise of that shared boundary can affect both engines. The GitHub mount exposes reviewed, named permission sets to agent identities. Those identities cannot call its unrestricted token endpoint or read engine configuration. The AWS mount exposes only the AWS roles each consumer needs. The AWS v0.3.0 release migrated to AWS SDK for Go v2. The upstream release warns that the large change carries extra regression risk, so production acceptance includes a real issuance, identity check, and revocation canary. The third-party GitHub plugin receives the same compatibility, issuance, and revocation gates.

​

Optional Slack OAuth proof of concept

The OpenBao role includes an isolated Slack OAuth proof of concept. It is turned off by default and does not change the active authentication path. When an operator deliberately enables it, the role requires an exact callback route. It requests only the bot scope needed to send messages, and gives its reader a short-lived, read-only identity. The reverse proxy may publish the exact callback path while the engine remains turned off. That route returns a no-content response and strips query parameters from access logs, so OAuth authorization data does not enter routine proxy logs.

​

Architecture

A single OpenBao service holds the truth; one lightweight agent per consumer renews leases and renders the secrets it needs into a memory-only file.

​

Auth model

• AppRole per service. Each consumer has a role ID (durable) and a secret ID (rotating). The Ansible role distributes role IDs in cleartext but obtains secret IDs from OpenBao, ideally as a response-wrapped, single-use, short-TTL token at provision time, so an intercepted bootstrap is self-evident.
• SSH CA for automation. Automation principals present a public key and receive a short-TTL signed certificate; the target host trusts the CA, not a pile of authorized_keys.
• Periodic tokens for long-running services where re-authentication would be operationally heavy.
• TLS cert-based auth for the most sensitive services as a defense-in-depth layer.

​

RBAC — split by resource domain

Access is split into one least-privilege AppRole per resource domain. A compromise of any single credential is scoped to that domain’s secrets, not the whole tree. Each AppRole is confined to its own path prefix. It can read (and, for the small set of identities that need to write, write) only the secrets its domain owns. The sharpest boundary is between the two IaC identities that touch the same underlying infrastructure secrets. One is scoped for apply: human-triggered and permitted to write. The other is scoped for plan: it runs potentially untrusted, VCS-driven plans, and is therefore strictly narrower. It is read-only, and confined to its own plan-time path, never the apply identity’s write scope. That asymmetry means a malicious plan cannot rewrite secrets a later apply step trusts. The same domain-isolation pattern repeats for every other consumer class. It covers configuration-management convergence, observability tooling, and self-hosted cloud services. It also covers monitoring/media apps, the local LLM serving stack, and AI agent identities (kept read-only and walled off from the infrastructure domains). Each has its own AppRole scoped to only the path prefix that consumer needs. It also gets a narrow break-glass identity for the vault’s own snapshot/backup operation. Each AppRole additionally carries a secret_id_bound_cidrs scoped to its consumer’s subnet, so even a leaked secret_id is unusable outside that consumer’s network segment.

​

Secret-zero — what bootstraps the bootstrapper

OpenBao cannot start sealed-and-empty on its own. Its secret-zero lives one tier up, in the Doppler bootstrap tier (T3). That tier holds the static seal key that auto-unseals the service, and the AppRole role_id/secret_id pairs that consumers present to authenticate. That material, plus the few values an external service reads directly, is all T3 holds. Everything else lives here or is minted by an engine on demand. The human-only recovery shares live in the Bitwarden vault (T4) as the break-glass path if both T2 and T3 are lost. On an always-on Mac agent host, the run-wrapper publishes each domain’s AppRole role_id/secret_id into the agent process’s environment, and publishes nothing else. It is promptless: an agent reads its AppRole as a plain environment value, never through an interactive unlock. The “zero prompts at 3 AM” property holds because no per-read gate sits in the path. Every other value the agent needs it fetches from here, with that AppRole.

​

Autonomous Agent API Authentication

Agents are provisioned with an OpenBao AppRole role_id and secret_id, placed in the agent environment by the run-wrapper. They must exchange these for a short-lived client_token via the OpenBao API. This happens before reading secrets or running pipelines that require OpenBao data, such as the infrastructure inventory. In practice, an agent exchanges its AppRole role_id/secret_id for that token by calling the vault’s standard AppRole login endpoint. It then uses the returned token for subsequent reads.

​

Boundary with the other tiers

• SOPS + age (T1): stays the source of truth for at-rest, in-repo config. OpenBao is for runtime-resolved secrets where lease/renew/rotation matters.
• Doppler (T3): holds secret-zero, and the few values an external service consumes without passing through here; OpenBao holds every other machine credential.
• Bitwarden (T4): human-only cold storage and OpenBao’s recovery shares.

The tiers compose; OpenBao is the primary machine path, not a replacement for the layers around it.

​

Sealed posture, audit trail, and recovery

• Sealed at rest. OpenBao restarts sealed. Routine restarts auto-unseal from the seal key held in the bootstrap tier; a full recovery from N-of-M shares is a Golden law #3 human-in-the-loop event.
• Audit log to Splunk. Every read, write, and policy change appears in Splunk via the standard OpenBao audit-device → file → Cribl Edge path.
• Raft-snapshot DR. An on-box systemd timer snapshots the active node on a schedule (leader-gated at runtime, authenticated with the least-privilege snapshot AppRole), integrity-checks each snapshot, and keeps them on a ZFS/PBS-backed volume that replicates off-box; a second copy to an encrypted S3 bucket is a tracked follow-up. Recovery is: restore the snapshot onto a fresh node, auto-unseal from secret-zero, and let the leases re-issue. Rehearsed with a drill on a non-production instance (Golden law #15).

​

See also

• SSH access: the certificate-first access ladder this CA implements, and why host-key checking is never turned off.
• Tools comparison: where T2 sits, and the bootstrap / read-path / DR flows drawn out.
• Agent secrets: the headless read path that makes OpenBao the primary interface.
• Golden laws: every preceding posture maps to a law.
• dryvist/ansible-proxmox-apps: the bring-up role.
• docs.dryvist.com: dryvist-internal specifics (topology, policy paths, lease TTLs) live there once this lands.