Monitoring agents

One picture: every collector, what it collects, where it runs, what it forwards to.

The monitoring stack runs across four hosts: the homelab Proxmox cluster, the macOS workstation, the OrbStack K8s cluster on that workstation, and the AWS-side Splunk install. Each tier picks the lowest-overhead path the host OS supports natively. This page maps every collector to a host and a destination. Every Edge feeds a Stream; only Stream reaches Splunk; the AWS tier is one routing-rule flip away. Detail per collector below.

​

The collectors

​

The Edge → Stream → Splunk invariant

Across every tier, the architectural rule is the same: Edge collects, Stream routes, Splunk indexes. Edge does not talk directly to Splunk anywhere on this stack. Stream is the only component with Splunk egress. On the OrbStack cluster this is enforced by network policies that lock Edge egress to Stream on HEC port 8088 only; on the homelab Proxmox side the rule is operational (single Stream tier), reinforced by firewall rules in tofu-proxmox/modules/firewall/. Two as-built details of that egress:

• Edge → Stream rides Cribl S2S as tcpjson on the dedicated cribl_s2s service port (a pipeline constant from the infrastructure repo, never hardcoded).
• Stream → Splunk is one HEC output per index, each with its own token. Tokens are derived, not distributed: UUIDv5 of splunk-hec-<index> in a shared private namespace, so Cribl and Splunk compute the same value independently. The namespace UUID is the only secret, and a leaked token scopes to one index. See the full family/port/index map.

​

Per-tool Cribl Edge packs

The AI-coding-tool packs sit on top of the OrbStack cluster’s cribl-edge-standalone: The macOS host telemetry pack cc-edge-the-mac-pack — renamed from cc-edge-macos-system — targets the native macOS Cribl Edge install (not the OrbStack-deployed one) — its exec inputs call macOS-only binaries that need host access, not a Linux container.

​

REST collectors

REST collectors run on Stream rather than Edge because they’re pull-based jobs against authenticated APIs — closer to the routing layer’s responsibility than the host-side capture layer.

​

OTel collectors

Anything inside the OrbStack cluster that speaks OTLP gets pointed at the in-cluster OTel collector. Anything outside the cluster but on the same Mac (Claude Code, Gemini Antigravity, VS Code) talks to the OTel collector through the NodePort or hands off to the Cribl Edge standalone pack.

​

Heartbeats

Four healthchecks.io CronJobs run in the OrbStack monitoring namespace as dead-man switches: A missed ping is the first signal of a broken pipeline — every other failure mode tends to be silent. Heartbeats prove the pipeline is up; per-index silence detectors prove the data is flowing. Every Splunk index carries an alert that fires when the index receives no events within its expected cadence, so a dead sender, port, or token surfaces as “index went quiet” rather than a gap discovered at search time.

​

Forwarding to AWS DR

The Splunk install behind cribl-stream is the homelab indexer. tofu-splunk-aws provisions the AWS-side DR footprint: same data shape, smaller indexer tier, ready to take a failover. Cribl Stream’s output config can be flipped from the homelab Splunk to the AWS HEC endpoint via a routing-rule change; downstream AI-observability dashboards keep working because they target the same indexes regardless of which Splunk tier is live.

​

See also