AI telemetry pipeline
Coral dashed edges are telemetry; solid green are the routing hops. Per-tool packs keep parsing isolated. A Copilot schema change doesn’t break the Claude pipeline. Diagram walkthroughs for every intake path live on Pipeline flows. That page covers each syslog family and AI command-line tool shipping. It also covers the LLM stack’s OTLP path, with separate NetFlow and HEC fan-in paths.Syslog intake families
Syslog intake is one port per source family, fronted by the ingress load balancers (HAProxy for TCP, nginx for UDP). Each family routes to its own Splunk index with a family sourcetype. A misbehaving sender is isolated at the port and visible at the index.
Port numbers are pipeline constants defined once in the infrastructure repo and surfaced to every consumer through the published inventory. No repo hardcodes them.
Two family notes:
- Managed Linux guests. Supported guests forward OS logs through the
linuxfamily. The syslog forwarder role owns the source group and destination mapping. - Hypervisor firewall drops ride the
linuxfamily: the node firewall logs every default-policy DROP to a file (never the journal), so each node’s rsyslog tails that file withimfileand forwards it alongside the journal stream. Searchable asindex=os process=pve-firewall. - The
macosfamily (521) is intentionally unused. BSD syslogd emits RFC3164 (no year, no timezone) and workstations deliberately keep local time, so a syslogd remote-forward can never be skew-safe. Macs ship through their local Cribl Edge instead (see below), whose sources carry absolute, TZ-qualified timestamps.
Timestamps: UTC at the source
Every host except the workstation Macs runs and logs UTC, enforced by configuration management, not assumed. This is doctrine, not preference. The pipeline’s config normalizer strips per-source timezone overrides on restart, so a source that stamps local time cannot be corrected downstream. A skewed source is fixed at the source. Skew is measured with index-time (eval skew=_indextime-_time), because event-time windows silently miss
mis-stamped events instead of surfacing them.
AI log ports (tcpjson)
AI-side logs skip syslog entirely. Cribl Edge tails the local log files and ships them as tcpjson over Cribl S2S to Cribl Stream, one port per source.
The Mac firewall leg tails the unified log (app firewall, network
extension, packet filter subsystems) as ndjson via a launch daemon. The
local Edge ships it with
index=firewall stamped at the source. Unified-log
timestamps carry a UTC offset, so a local-time workstation stays skew-safe.
Edge-to-Stream relay traffic itself rides Cribl S2S tcpjson on the dedicated cribl_s2s service port from the same pipeline constants.
Per-index HEC outputs
Only Cribl Stream talks to Splunk. It uses one HEC output per index. Each output has a token scoped to one index. Each index also has a silence detector, a per-index alert that fires when the index stops receiving events within its expected cadence. A broken port, token, or sender surfaces as “index went quiet” instead of a silent gap discovered weeks later. Detectors on structurally empty indexes ship off (currentlynetflow, whose gateway never exports despite the receive chain being
armed). An alert that fires on every cycle only trains fatigue.
Splunk apps
The AI-observability apps and TAs live in a separate organization’s repositories. Two pieces matter on this site:
Where to check it live:
- Live: your Splunk instance’s Dashboard Studio, and its silence-detector alerts under Settings → Searches, reports, and alerts.
- Config: the Dashboards app and TA repositories.
- Owner doc: ansible-splunk for the indexes and HEC tokens these dashboards read.
Reusable observability content
Cribl pipeline and route logic lives in function-specific Cribl packs. Splunk apps and knowledge objects live in function-specific app repositories. Ansible carries environment wiring, including output destinations and endpoint details. It also carries credential references and pack or app pins. Each Cribl pack lives in one function-specific repository.Cribl Stream collectors
Why per-tool packs
Each AI coding tool emits slightly different telemetry shapes. Per-tool packs keep the parsing and enrichment isolated; a Copilot schema change shouldn’t break the Claude pipeline. The shared CIM mapping in the TA is where the normalization happens.Where Splunk runs
tofu-splunk-aws
OpenTofu for the AWS-side Splunk footprint. VPC, KMS, EC2, IAM, DR-ready.
ansible-splunk
The configuration tier. Splunk install, indexes, HEC tokens, storage tiering.
Where to go next
Mac Pack
The macOS host telemetry pack: unified logs, system metrics, power.
Monitoring agents
Cross-stack map of every collector and where it runs.
Data pipelines
Log and NetFlow intake, the non-AI side of the pipeline.
Configuration
Ansible playbooks that deploy the Cribl tier this pipeline runs on.