Pipeline flows

One overview, six zooms. Every source, every port, every index — and which legs are live versus planned.

Every log and metric in this homelab converges on the same spine: sources reach an ingress load balancer or a local Cribl Edge, Cribl Stream routes, and Splunk indexes over per-index HEC. The diagrams below zoom into each ingest path. Flows that are not live yet carry a planned badge. The family/port/index tables at the end cover every syslog family and AI log port.

​

Monitoring overview

Five source groups, two ingress load balancers (HAProxy for TCP syslog, nginx for UDP), one Cribl Edge/Stream spine, two sinks: Splunk (per-index HEC) and Langfuse (OTLP traces). Solid green edges are the network/ingress hops; dashed coral edges carry telemetry. Everything below zooms into one leg of this picture.

​

Syslog families

Each syslog source family gets its own ingress port (514–523, plus the high port 1514 for UniFi), so a misbehaving sender is isolated at the port and each family routes to its own index. Ports come from the pipeline constants in the infrastructure repo — never hardcoded. The full family → port → index → sourcetype table lives on the observability overview. Hypervisor firewall drops ride the linux family: the node firewall logs to a file rather than the journal, so each node’s rsyslog tails it with imfile and forwards it on the same port — searchable as index=os process=pve-firewall.

​

AI-CLI log shipping

AI CLI logs never touch syslog. Each CLI writes local JSONL log files; the MacBook’s standalone Cribl Edge tails them with file inputs and ships them as tcpjson (one port per CLI, 10311–10315) straight to Cribl Stream, which lands them in per-index HEC outputs. The same Edge also ships the Mac firewall unified-log: a launch daemon tails the firewall subsystems (application firewall, network extension, packet filter) as ndjson into a rotated file, and a file input forwards it over the shared S2S port with index=firewall stamped at the source. This replaced the retired BSD-syslogd remote forward, which could never be skew-safe (RFC3164 carries no year and no timezone, and workstations deliberately keep local time).

​

Mac Studio LLM stack

Local inference on the Mac Studio: clients hit the caddy gate on :11434, which fronts llama-swap, which spins model workers up and down. Logs and metrics from all three layers leave through the host’s Cribl Edge on tcpjson ports 10321–10323.

​

OTLP dual-write

Orchestration apps emit OpenTelemetry to Cribl Edge’s OTLP HTTP source — never to a backend directly. Edge fans out: traces to Langfuse (live), everything to Splunk (planned — the leg is wired but not yet enabled).

​

NetFlow (gateway export gap — parked)

Flow records from the gateway and switches would take the UDP side of the ingress tier — and the whole receive chain (UDP LB, Edge NetFlow pipeline, dedicated index) is built, armed, and idle. The gateway shows NetFlow enabled in its console but emits zero IPFIX packets, a gateway-side export gap. The path stays documented and dotted; the index’s silence detector ships disabled so a structurally-empty index doesn’t alert on every cycle. Desired state (including the export settings the console claims) is version-controlled in the network IaC repo.

​

HEC fan-in

Cribl Stream is the only component with Splunk egress, and it fans out into one HEC output per index. Every output carries its own token, derived as a UUIDv5 of splunk-hec-\<index\> in a shared private namespace — Cribl and Splunk compute the same token independently, and the namespace UUID is the only secret. Every index also carries a silence detector — an alert that fires when the index goes quiet.

​

Source → port → index map

Every syslog family and every AI log port, in one table. Syslog rides the HAProxy TCP LB (UDP variants via nginx); AI log ports are tcpjson from Cribl Edge to Cribl Stream.

​

The network underneath

The pipeline rides the trust-ordered VLAN tier model: VLAN tag = tier × 10, subnets follow the 192.168.\<vlan-tag\>.0/24 placeholder pattern (real subnets are injected at runtime, never committed). The observability tier (VLAN 40) hosts the ingress LBs, Cribl, and Splunk; every other tier is a log source. The full VMID ↔ VLAN convention — how a guest’s six-digit ID encodes its tier — is on VMID & network tier model.

​

See also