Pipeline flows

One overview, six zooms. Every source, every port, every index: this page shows which legs are live versus planned.

Every log and metric in this homelab converges on the same spine. Sources reach an ingress load balancer or a local Cribl Edge. Cribl Stream then routes each log, and Splunk indexes it over per-index HEC. The diagrams below zoom into each load path. Flows that are not live yet carry a planned badge. The family/port/index tables at the end cover every syslog family and AI log port.

​

Monitoring overview

Five source groups, two ingress load balancers (HAProxy for TCP syslog, nginx for UDP), one Cribl Edge/Stream spine, three sinks. Splunk takes everything over per-index HEC; Langfuse and Phoenix each take the OTLP traces. Solid green edges are the network/ingress hops; dashed coral edges carry telemetry. Everything below zooms into one leg of this picture.

​

Syslog families

Each syslog source family gets its own ingress port (514–523, plus the high port 1514 for UniFi). This isolates a misbehaving sender at the port, and each family routes to its own index. Ports come from the pipeline constants in the infrastructure repo; the pipeline never hardcodes them. The full family → port → index → sourcetype table lives on the observability overview. Hypervisor firewall drops ride the linux family. The node firewall logs to a file rather than the journal. So each node’s rsyslog tails it with imfile and forwards it on the same port. Search for it as index=os process=pve-firewall.

​

AI command-line log shipping

AI command-line tool logs never reach syslog. Each command-line tool writes local JSONL log files. The MacBook’s standalone Cribl Edge tails them with file inputs and ships them as tcpjson (one port per command-line tool, 10311–10315). Cribl Stream then lands them in per-index HEC outputs. The same Edge also ships the Mac firewall unified-log. A launch daemon tails the firewall subsystems (application firewall, network extension, packet filter) as ndjson into a rotated file. A file input then forwards it over the shared S2S port with index=firewall stamped at the source. The pipeline replaced the retired BSD-syslogd remote forward, which could never be skew-safe. RFC3164 carries no year and no timezone, and workstations deliberately keep local time.

​

Mac Studio LLM stack

Local inference on the Mac Studio: clients hit the caddy gate on :11434, which fronts llama-swap, which spins model workers up and down. Logs and metrics from all three layers leave through the host’s Cribl Edge on tcpjson ports 10321–10323.

​

OTLP fan-out

Orchestration apps emit OpenTelemetry to Cribl Edge’s OTLP HTTP source. They never emit it to a backend directly. Edge fans traces out to Langfuse and Phoenix — never Splunk, which does not fit the trace format. See Data pipelines for the fan-out default this follows.

​

NetFlow (gateway export gap: parked)

Flow records from the gateway and switches would take the UDP side of the ingress tier. The whole receive chain (UDP LB, Edge NetFlow pipeline, dedicated index) is built, armed, and idle. The gateway shows NetFlow enabled in its console but emits zero IPFIX packets, a gateway-side export gap. The path stays documented and dotted. The index’s silence detector ships turned off, so a structurally empty index doesn’t alert on every cycle. Desired state (including the export settings the console claims) is version-controlled in the network IaC repo.

​

HEC fan-in

Cribl Stream is the only component with Splunk egress, and it fans out into one HEC output per index. Every output carries its own token, derived as a UUIDv5 of splunk-hec-\<index\> in a shared private namespace. Cribl and Splunk compute the same token independently, and the namespace UUID is the only secret. Every index also carries a silence detector. It is an alert that fires when the index goes quiet.

​

Source → port → index map

Every syslog family and every AI log port, in one table. Syslog rides the HAProxy TCP LB (UDP variants via nginx); AI log ports are tcpjson from Cribl Edge to Cribl Stream.

​

The network underneath

The pipeline rides the trust-ordered VLAN tier model. The VLAN tag = tier × 10, and subnets follow the 192.168.\<vlan-tag\>.0/24 placeholder pattern (real subnets are injected at runtime, never committed). The observability tier (VLAN 40) hosts the ingress LBs, Cribl, and Splunk. Every other tier is a log source. The full VMID ↔ VLAN convention is on VMID & network tier model. It covers how a guest’s six-digit ID encodes its tier.

​

See also