Observability

If an AI agent touched code, there’s a trace.

Every AI coding interaction emits OpenTelemetry — IDE events, model calls, token counts, latency, cost. The Cribl Edge tier collects and reshapes it. Cribl Stream routes it. Splunk indexes it. Purpose-built Splunk apps make it readable.

​

AI telemetry pipeline

Coral dashed edges are telemetry; solid green are the routing hops. Per-tool packs keep parsing isolated — a Copilot schema change doesn’t break the Claude pipeline. Diagram walkthroughs for every ingest path — syslog families, AI-CLI shipping, the LLM stack, OTLP dual-write, NetFlow, HEC fan-in — live on Pipeline flows.

​

Syslog ingest families

Syslog ingest is one port per source family, fronted by the ingress load balancers (HAProxy for TCP, nginx for UDP). Each family routes to its own Splunk index with a family sourcetype, so a misbehaving sender is isolated at the port and visible at the index. Port numbers are pipeline constants defined once in the infrastructure repo and surfaced to every consumer through the published inventory — no repo hardcodes them. Two family notes:

• Hypervisor firewall drops ride the linux family: the node firewall logs every default-policy DROP to a file (never the journal), so each node’s rsyslog tails that file with imfile and forwards it alongside the journal stream. Searchable as index=os process=pve-firewall.
• The macos family (521) is intentionally unused. BSD syslogd emits RFC3164 (no year, no timezone) and workstations deliberately keep local time, so a syslogd remote-forward can never be skew-safe. Macs ship through their local Cribl Edge instead (see below), whose sources carry absolute, TZ-qualified timestamps.

​

Timestamps: UTC at the source

Every host except the workstation Macs runs and logs UTC — enforced by configuration management, not assumed. This is doctrine, not preference: the pipeline’s config normalizer strips per-source timezone overrides on restart, so a source that stamps local time cannot be corrected downstream. A skewed source is fixed at the source. Skew is measured with index-time (eval skew=_indextime-_time), because event-time windows silently miss mis-stamped events instead of surfacing them.

​

AI log ports (tcpjson)

AI-side logs skip syslog entirely: Cribl Edge tails the local log files and ships them as tcpjson over Cribl S2S to Cribl Stream, one port per source. The Mac firewall leg tails the unified log (application firewall, network extension, packet filter subsystems) as ndjson via a launch daemon, and the local Edge ships it with index=firewall stamped at the source — unified-log timestamps carry a UTC offset, so a local-time workstation stays skew-safe. Edge-to-Stream relay traffic itself rides Cribl S2S tcpjson on the dedicated cribl_s2s service port from the same pipeline constants.

​

Per-index HEC outputs and derived tokens

Cribl Stream is the only component that talks to Splunk, and it does so through one HEC output per index, each with its own token. Tokens are never distributed: both sides derive the same value as a UUIDv5 of splunk-hec-<index> in a shared private namespace, so Cribl outputs and Splunk HEC inputs agree by construction. The namespace UUID is the only secret; a leaked token scopes to a single index. Each index also has a silence detector — a per-index alert that fires when the index stops receiving events within its expected cadence. A broken port, token, or sender surfaces as “index went quiet” instead of a silent gap discovered weeks later. Detectors on structurally-empty indexes ship disabled (currently netflow, whose gateway never exports despite the receive chain being armed) — an alert that fires on every cycle only trains fatigue.

​

Splunk apps

The AI-observability apps and TAs live under a separate organization at github.com/visicore. Two pieces matter on this site:

​

Cribl Edge packs (collectors)

​

Cribl Stream collectors

​

Why per-tool packs

Each AI coding tool emits slightly different telemetry shapes. Per-tool packs keep the parsing and enrichment isolated; a Copilot schema change shouldn’t break the Claude pipeline. The shared CIM mapping in the TA is where the normalization happens.

​

Where Splunk runs

​

Where to go next