Skip to main content
If an AI agent touched code, there’s a trace. Every AI coding interaction emits OpenTelemetry. It covers IDE events and model calls. It also records token usage. The telemetry includes latency and cost. The Cribl Edge tier collects and reshapes it. Cribl Stream routes it. Splunk indexes it. Purpose-built Splunk apps make it readable.

AI telemetry pipeline

Coral dashed edges are telemetry; solid green are the routing hops. Per-tool packs keep parsing isolated. A Copilot schema change doesn’t break the Claude pipeline. Diagram walkthroughs for every intake path live on Pipeline flows. That page covers each syslog family and AI command-line tool shipping. It also covers the LLM stack’s OTLP path, with separate NetFlow and HEC fan-in paths.

Syslog intake families

Syslog intake is one port per source family, fronted by the ingress load balancers (HAProxy for TCP, nginx for UDP). Each family routes to its own Splunk index with a family sourcetype. A misbehaving sender is isolated at the port and visible at the index. Port numbers are pipeline constants defined once in the infrastructure repo and surfaced to every consumer through the published inventory. No repo hardcodes them. Two family notes:
  • Managed Linux guests. Supported guests forward OS logs through the linux family. The syslog forwarder role owns the source group and destination mapping.
  • Hypervisor firewall drops ride the linux family: the node firewall logs every default-policy DROP to a file (never the journal), so each node’s rsyslog tails that file with imfile and forwards it alongside the journal stream. Searchable as index=os process=pve-firewall.
  • The macos family (521) is intentionally unused. BSD syslogd emits RFC3164 (no year, no timezone) and workstations deliberately keep local time, so a syslogd remote-forward can never be skew-safe. Macs ship through their local Cribl Edge instead (see below), whose sources carry absolute, TZ-qualified timestamps.

Timestamps: UTC at the source

Every host except the workstation Macs runs and logs UTC, enforced by configuration management, not assumed. This is doctrine, not preference. The pipeline’s config normalizer strips per-source timezone overrides on restart, so a source that stamps local time cannot be corrected downstream. A skewed source is fixed at the source. Skew is measured with index-time (eval skew=_indextime-_time), because event-time windows silently miss mis-stamped events instead of surfacing them.

AI log ports (tcpjson)

AI-side logs skip syslog entirely. Cribl Edge tails the local log files and ships them as tcpjson over Cribl S2S to Cribl Stream, one port per source. The Mac firewall leg tails the unified log (app firewall, network extension, packet filter subsystems) as ndjson via a launch daemon. The local Edge ships it with index=firewall stamped at the source. Unified-log timestamps carry a UTC offset, so a local-time workstation stays skew-safe. Edge-to-Stream relay traffic itself rides Cribl S2S tcpjson on the dedicated cribl_s2s service port from the same pipeline constants.

Per-index HEC outputs

Only Cribl Stream talks to Splunk. It uses one HEC output per index. Each output has a token scoped to one index. Each index also has a silence detector, a per-index alert that fires when the index stops receiving events within its expected cadence. A broken port, token, or sender surfaces as “index went quiet” instead of a silent gap discovered weeks later. Detectors on structurally empty indexes ship off (currently netflow, whose gateway never exports despite the receive chain being armed). An alert that fires on every cycle only trains fatigue.

Splunk apps

The AI-observability apps and TAs live in a separate organization’s repositories. Two pieces matter on this site: Where to check it live:
  • Live: your Splunk instance’s Dashboard Studio, and its silence-detector alerts under Settings → Searches, reports, and alerts.
  • Config: the Dashboards app and TA repositories.
  • Owner doc: ansible-splunk for the indexes and HEC tokens these dashboards read.

Reusable observability content

Cribl pipeline and route logic lives in function-specific Cribl packs. Splunk apps and knowledge objects live in function-specific app repositories. Ansible carries environment wiring, including output destinations and endpoint details. It also carries credential references and pack or app pins. Each Cribl pack lives in one function-specific repository.

Cribl Stream collectors

Why per-tool packs

Each AI coding tool emits slightly different telemetry shapes. Per-tool packs keep the parsing and enrichment isolated; a Copilot schema change shouldn’t break the Claude pipeline. The shared CIM mapping in the TA is where the normalization happens.

Where Splunk runs

tofu-splunk-aws

OpenTofu for the AWS-side Splunk footprint. VPC, KMS, EC2, IAM, DR-ready.

ansible-splunk

The configuration tier. Splunk install, indexes, HEC tokens, storage tiering.

Where to go next

Mac Pack

The macOS host telemetry pack: unified logs, system metrics, power.

Monitoring agents

Cross-stack map of every collector and where it runs.

Data pipelines

Log and NetFlow intake, the non-AI side of the pipeline.

Configuration

Ansible playbooks that deploy the Cribl tier this pipeline runs on.