> ## Documentation Index > Fetch the complete documentation index at: https://docs.jacobpevans.com/llms.txt > Use this file to discover all available pages before exploring further. # Pipeline flows > The monitoring pipeline as diagrams: syslog families, AI-CLI shipping, the local LLM stack, OTLP dual-write, NetFlow, and the per-index HEC fan-in — one zoom per concern. > One overview, six zooms. Every source, every port, every index — and which legs are live versus planned. Every log and metric in this homelab converges on the same spine: sources reach an ingress load balancer or a local Cribl Edge, Cribl Stream routes, and Splunk indexes over per-index HEC. The diagrams below zoom into each ingest path. Flows that are not live yet carry a **planned** badge. The [family/port/index tables](#source--port--index-map) at the end cover every syslog family and AI log port. ## Monitoring overview Five source groups, two ingress load balancers (HAProxy for TCP syslog, nginx for UDP), one Cribl Edge/Stream spine, two sinks: Splunk (per-index HEC) and Langfuse (OTLP traces). {/* Ranks: 5 sources max. Total nodes: 11 (≤12). Aspect: ~2:1 LR. Pass. */} ```mermaid theme={null} %%{init: {'theme':'base','look':'handDrawn','themeVariables':{'fontFamily':'Geist','fontSize':'14px','primaryColor':'#102937','primaryTextColor':'#F4EFE6','primaryBorderColor':'#4FB3A9','lineColor':'#4FB3A9','secondaryColor':'#0B1D2A','tertiaryColor':'#1A2A38','clusterBkg':'rgba(79,179,169,0.08)','clusterBorder':'#4FB3A9'}}}%% flowchart LR Gear([fa:fa-network-wired Network gear]) Hosts([fa:fa-server Linux · Windows · Mac]) CLIs([fa:fa-robot AI CLIs]) LLM([fa:fa-microchip LLM stacks]) Bao([fa:fa-vault OpenBao audit]) HAP([HAProxy TCP LB]) NGX([nginx UDP LB]) Edge([Cribl Edge]) Stream([Cribl Stream]) SP[(Splunk
per-index HEC)] LF([Langfuse]) Gear --> HAP Gear --> NGX Hosts --> HAP HAP --> Edge NGX --> Edge CLIs --> Edge LLM --> Edge Bao --> Edge Edge --> Stream Stream --> SP Stream --> LF classDef src fill:#102937,stroke:#E06B4A,stroke-width:2px,color:#F4EFE6; classDef ai fill:#102937,stroke:#E06B4A,stroke-width:2px,color:#F4EFE6; classDef hop fill:#102937,stroke:#4FB3A9,stroke-width:2px,color:#F4EFE6; classDef sink fill:#102937,stroke:#F4EFE6,stroke-width:2px,color:#F4EFE6; class Gear,Hosts,Bao src class CLIs,LLM ai class HAP,NGX,Edge,Stream hop class SP,LF sink click Edge "/observability/monitoring-agents" "Every collector and where it runs" click SP "/observability/repos/ansible-splunk" "Splunk install + indexes" click LF "/observability/llm-observability" "Langfuse and the OTLP path" linkStyle 0,1,2,3,4 stroke:#4FB3A9,stroke-width:2px; linkStyle 5,6,7,8,9,10 stroke:#E06B4A,stroke-width:2px,stroke-dasharray:4 3; ``` Solid green edges are the network/ingress hops; dashed coral edges carry telemetry. Everything below zooms into one leg of this picture. ## Syslog families Each syslog source family gets its own ingress port (514–523, plus the high port 1514 for UniFi), so a misbehaving sender is isolated at the port and each family routes to its own index. Ports come from the pipeline constants in the infrastructure repo — never hardcoded. {/* Ranks: 5 sources max. Total nodes: 8 (≤12). Aspect: ~2.5:1 LR. Pass. */} ```mermaid theme={null} %%{init: {'theme':'base','look':'handDrawn','themeVariables':{'fontFamily':'Geist','fontSize':'14px','primaryColor':'#102937','primaryTextColor':'#F4EFE6','primaryBorderColor':'#4FB3A9','lineColor':'#4FB3A9','secondaryColor':'#0B1D2A','tertiaryColor':'#1A2A38','clusterBkg':'rgba(79,179,169,0.08)','clusterBorder':'#4FB3A9'}}}%% flowchart LR U([UniFi · 514 · 1514 · 520]) FW([Palo Alto · ASA
515 · 516]) OS([Linux · Windows · macOS
517 · 518 · 521]) HP([Honeypots · 519]) DQ([DNS query · proxy
522 · 523]) HAP([HAProxy
514–523]) Edge([Cribl Edge
syslog pipelines]) SP[(Splunk
per-index HEC)] U --> HAP FW --> HAP OS --> HAP HP --> HAP DQ --> HAP HAP --> Edge Edge --> SP classDef src fill:#102937,stroke:#E06B4A,stroke-width:2px,color:#F4EFE6; classDef hop fill:#102937,stroke:#4FB3A9,stroke-width:2px,color:#F4EFE6; classDef sink fill:#102937,stroke:#F4EFE6,stroke-width:2px,color:#F4EFE6; class U,FW,OS,HP,DQ src class HAP,Edge hop class SP sink click SP "/observability/repos/ansible-splunk" "Splunk install + indexes" linkStyle 0,1,2,3,4 stroke:#4FB3A9,stroke-width:2px; linkStyle 5,6 stroke:#E06B4A,stroke-width:2px,stroke-dasharray:4 3; ``` The full family → port → index → sourcetype table lives on the [observability overview](/observability/overview#syslog-ingest-families). ## AI-CLI log shipping AI CLI logs never touch syslog. Each CLI writes local JSONL log files; the MacBook's standalone Cribl Edge tails them with file inputs and ships them as tcpjson (one port per CLI, 10311–10315) straight to Cribl Stream, which lands them in per-index HEC outputs. {/* Ranks: ≤3 in subgraph. Total nodes: 5 (≤12). Aspect: ~3:1 LR. Pass. */} ```mermaid theme={null} %%{init: {'theme':'base','look':'handDrawn','themeVariables':{'fontFamily':'Geist','fontSize':'14px','primaryColor':'#102937','primaryTextColor':'#F4EFE6','primaryBorderColor':'#4FB3A9','lineColor':'#4FB3A9','secondaryColor':'#0B1D2A','tertiaryColor':'#1A2A38','clusterBkg':'rgba(79,179,169,0.08)','clusterBorder':'#4FB3A9'}}}%% flowchart LR subgraph mb["MacBook"] CLI([fa:fa-robot AI CLIs]) Files[(JSONL logs)] Edge([Cribl Edge
file inputs]) end Stream([Cribl Stream]) SP[(Splunk
per-index HEC)] CLI --> Files Files --> Edge Edge -->|tcpjson 10311–10315| Stream Stream --> SP classDef ai fill:#102937,stroke:#E06B4A,stroke-width:2px,color:#F4EFE6; classDef hop fill:#102937,stroke:#4FB3A9,stroke-width:2px,color:#F4EFE6; classDef sink fill:#102937,stroke:#F4EFE6,stroke-width:2px,color:#F4EFE6; class CLI ai class Files,Edge,Stream hop class SP sink click Edge "/observability/repos/cc-edge-the-mac-pack" "The standalone Mac Edge install" linkStyle 0,1,2,3 stroke:#E06B4A,stroke-width:2px,stroke-dasharray:4 3; ``` ## Mac Studio LLM stack Local inference on the Mac Studio: clients hit the caddy gate on `:11434`, which fronts llama-swap, which spins model workers up and down. Logs and metrics from all three layers leave through the host's Cribl Edge on tcpjson ports 10321–10323. {/* Ranks: ≤2 siblings. Subgraph: 5 nodes (cap). Total: 7 (≤12). Aspect: ~2:1 LR. Pass. */} ```mermaid theme={null} %%{init: {'theme':'base','look':'handDrawn','themeVariables':{'fontFamily':'Geist','fontSize':'14px','primaryColor':'#102937','primaryTextColor':'#F4EFE6','primaryBorderColor':'#4FB3A9','lineColor':'#4FB3A9','secondaryColor':'#0B1D2A','tertiaryColor':'#1A2A38','clusterBkg':'rgba(79,179,169,0.08)','clusterBorder':'#4FB3A9'}}}%% flowchart LR Client((Clients)) subgraph ms["Mac Studio"] Gate{"caddy gate
:11434"} Swap([llama-swap]) W1([worker · model A]) W2([worker · model B]) Edge([Cribl Edge]) end Stream([Cribl Stream]) Client --> Gate Gate --> Swap Swap --> W1 Swap --> W2 Gate -.-> Edge W1 -.-> Edge W2 -.-> Edge Edge -->|tcpjson 10321–10323| Stream classDef external fill:#102937,stroke:#E6B35A,stroke-width:2px,color:#F4EFE6; classDef gate fill:#102937,stroke:#E06B4A,stroke-width:2.5px,color:#F4EFE6; classDef ai fill:#102937,stroke:#E06B4A,stroke-width:2px,color:#F4EFE6; classDef hop fill:#102937,stroke:#4FB3A9,stroke-width:2px,color:#F4EFE6; class Client external class Gate gate class W1,W2 ai class Swap,Edge,Stream hop click Stream "/observability/monitoring-agents" "Every collector and where it runs" linkStyle 0,1,2,3 stroke:#4FB3A9,stroke-width:2px; linkStyle 4,5,6,7 stroke:#E06B4A,stroke-width:2px,stroke-dasharray:4 3; ``` ## OTLP dual-write Orchestration apps emit OpenTelemetry to Cribl Edge's OTLP HTTP source — never to a backend directly. Edge fans out: traces to Langfuse (live), everything to Splunk (**planned** — the leg is wired but not yet enabled). {/* Ranks: ≤2. Total nodes: 4 (≤12). Aspect: ~3:1 LR. Pass. */} ```mermaid theme={null} %%{init: {'theme':'base','look':'handDrawn','themeVariables':{'fontFamily':'Geist','fontSize':'14px','primaryColor':'#102937','primaryTextColor':'#F4EFE6','primaryBorderColor':'#4FB3A9','lineColor':'#4FB3A9','secondaryColor':'#0B1D2A','tertiaryColor':'#1A2A38','clusterBkg':'rgba(79,179,169,0.08)','clusterBorder':'#4FB3A9'}}}%% flowchart LR Apps([Orchestration apps
OpenLLMetry]) Edge([Cribl Edge
OTLP HTTP]) LF([Langfuse]) SP[(Splunk · planned)] Apps --> Edge Edge -->|traces| LF Edge -.planned.-> SP classDef ai fill:#102937,stroke:#E06B4A,stroke-width:2px,color:#F4EFE6; classDef hop fill:#102937,stroke:#4FB3A9,stroke-width:2px,color:#F4EFE6; classDef sink fill:#102937,stroke:#F4EFE6,stroke-width:2px,color:#F4EFE6; class Apps ai class Edge hop class LF,SP sink click LF "/observability/llm-observability" "Langfuse and the OTLP path" linkStyle 0,1 stroke:#E06B4A,stroke-width:2px,stroke-dasharray:4 3; linkStyle 2 stroke:#E6B35A,stroke-width:1.5px,stroke-dasharray:2 4; ``` ## NetFlow *(planned — being revived)* Flow records from the gateway and switches take the UDP side of the ingress tier. The whole path is **planned / being revived** — amber dotted end to end until it ships again. {/* Ranks: 1 per column. Total nodes: 4 (≤12). Aspect: ~4:1 LR (short chain). Pass. */} ```mermaid theme={null} %%{init: {'theme':'base','look':'handDrawn','themeVariables':{'fontFamily':'Geist','fontSize':'14px','primaryColor':'#102937','primaryTextColor':'#F4EFE6','primaryBorderColor':'#4FB3A9','lineColor':'#4FB3A9','secondaryColor':'#0B1D2A','tertiaryColor':'#1A2A38','clusterBkg':'rgba(79,179,169,0.08)','clusterBorder':'#4FB3A9'}}}%% flowchart LR Gear([fa:fa-network-wired Gateway · switches]) NGX([nginx UDP LB]) Edge([Cribl Edge
NetFlow pipeline]) SP[(Splunk)] Gear -.planned.-> NGX NGX -.-> Edge Edge -.-> SP classDef src fill:#102937,stroke:#E06B4A,stroke-width:2px,color:#F4EFE6; classDef hop fill:#102937,stroke:#4FB3A9,stroke-width:2px,color:#F4EFE6; classDef sink fill:#102937,stroke:#F4EFE6,stroke-width:2px,color:#F4EFE6; class Gear src class NGX,Edge hop class SP sink click SP "/observability/repos/ansible-splunk" "Splunk install + indexes" linkStyle 0,1,2 stroke:#E6B35A,stroke-width:1.5px,stroke-dasharray:2 4; ``` ## HEC fan-in Cribl Stream is the only component with Splunk egress, and it fans out into **one HEC output per index**. Every output carries its own token, derived as a UUIDv5 of `splunk-hec-\` in a shared private namespace — Cribl and Splunk compute the same token independently, and the namespace UUID is the only secret. {/* Subgraphs: 4 + 3 nodes (≤5 each). Total nodes: 8 (≤12). Aspect: ~2:1 LR. Pass. */} ```mermaid theme={null} %%{init: {'theme':'base','look':'handDrawn','themeVariables':{'fontFamily':'Geist','fontSize':'14px','primaryColor':'#102937','primaryTextColor':'#F4EFE6','primaryBorderColor':'#4FB3A9','lineColor':'#4FB3A9','secondaryColor':'#0B1D2A','tertiaryColor':'#1A2A38','clusterBkg':'rgba(79,179,169,0.08)','clusterBorder':'#4FB3A9'}}}%% flowchart LR Stream([Cribl Stream
uuid5 token per index]) subgraph net["Network indexes"] direction TB Iu[(unifi)] If[(firewall)] Id[(dns)] Ip[(proxy)] end subgraph hostai["Host & AI indexes"] direction TB Io[(os)] Ih[(honeypot)] Ia[(AI log indexes)] end Stream -->|per-index HEC| Iu Iu ~~~ If If ~~~ Id Id ~~~ Ip Stream -->|per-index HEC| Io Io ~~~ Ih Ih ~~~ Ia classDef hop fill:#102937,stroke:#4FB3A9,stroke-width:2px,color:#F4EFE6; classDef sink fill:#102937,stroke:#F4EFE6,stroke-width:2px,color:#F4EFE6; class Stream hop class Iu,If,Id,Ip,Io,Ih,Ia sink click Stream "/observability/monitoring-agents" "Stream is the only Splunk egress" linkStyle 0,4 stroke:#E06B4A,stroke-width:2px,stroke-dasharray:4 3; ``` Every index also carries a [silence detector](/observability/overview#per-index-hec-outputs-and-derived-tokens) — an alert that fires when the index goes quiet. ## Source → port → index map Every syslog family and every AI log port, in one table. Syslog rides the HAProxy TCP LB (UDP variants via nginx); AI log ports are tcpjson from Cribl Edge to Cribl Stream. | Source | Family | Port(s) | Index | | ----------------------------------- | ----------- | ----------------------- | --------------------- | | UniFi gear | `unifi` | 514 (std) / 1514 (high) | `unifi` | | Palo Alto firewall | `palo_alto` | 515 | `firewall` | | Cisco ASA | `cisco_asa` | 516 | `firewall` | | Linux hosts | `linux` | 517 | `os` | | Windows hosts | `windows` | 518 | `os` | | Honeypots | `honeypot` | 519 | `honeypot` | | UniFi firewall logs | `unifi_fw` | 520 | `firewall` | | macOS hosts | `macos` | 521 | `os` | | DNS query logs | `dns_query` | 522 | `dns` | | Forward proxy | `proxy` | 523 | `proxy` | | AI CLIs (one port per CLI) | `ai_log` | 10311–10315 | AI CLI log indexes | | LLM stack (gate · router · workers) | `ai_log` | 10321–10323 | LLM stack log indexes | | OpenBao | `ai_log` | 10331 | OpenBao audit index | ## The network underneath The pipeline rides the trust-ordered VLAN tier model: VLAN tag = tier × 10, subnets follow the `192.168.\.0/24` placeholder pattern (real subnets are injected at runtime, never committed). The observability tier (VLAN 40) hosts the ingress LBs, Cribl, and Splunk; every other tier is a log source. {/* Subgraphs: 5 + 4 nodes (≤5 each). Total nodes: 10 (≤12). Aspect: ~2:1 LR. Pass. */} ```mermaid theme={null} %%{init: {'theme':'base','look':'handDrawn','themeVariables':{'fontFamily':'Geist','fontSize':'14px','primaryColor':'#102937','primaryTextColor':'#F4EFE6','primaryBorderColor':'#4FB3A9','lineColor':'#4FB3A9','secondaryColor':'#0B1D2A','tertiaryColor':'#1A2A38','clusterBkg':'rgba(79,179,169,0.08)','clusterBorder':'#4FB3A9'}}}%% flowchart LR GW((UniFi
gateway)) subgraph upper["Tiers 10–50 · most trusted"] direction TB T1([10 · Core services]) T2([20 · Storage]) T3([30 · Data / compute]) T4([40 · Observability]) T5([50 · AI / ML]) end subgraph lower["Tiers 60–90 · least trusted"] direction TB T6([60 · Applications]) T7([70 · Media]) T8([80 · Home / IoT]) T9([90 · Untrusted / guest]) end GW --> T1 T1 ~~~ T2 T2 ~~~ T3 T3 ~~~ T4 T4 ~~~ T5 GW --> T6 T6 ~~~ T7 T7 ~~~ T8 T8 ~~~ T9 classDef external fill:#102937,stroke:#E6B35A,stroke-width:2px,color:#F4EFE6; classDef host fill:#102937,stroke:#4FB3A9,stroke-width:2px,color:#F4EFE6; classDef gate fill:#102937,stroke:#E06B4A,stroke-width:2.5px,color:#F4EFE6; class GW external class T1,T2,T3,T5,T6,T7,T8,T9 host class T4 gate click T4 "/observability/monitoring-agents" "The observability tier's collectors" linkStyle 0,5 stroke:#4FB3A9,stroke-width:2px; ``` The full VMID ↔ VLAN convention — how a guest's six-digit ID encodes its tier — is on [VMID & network tier model](/infrastructure/vmid-network-tiers). ## See also The family/port/index tables and the AI telemetry pipeline. Every collector, what it collects, where it runs. The OTLP → Langfuse + Splunk dual-write in depth. The original log/NetFlow architecture page.