LLM observability

If a model was called, there’s a trace. You can see what it cost. The AI-coding-tool pipeline traces the IDEs. This is its sibling for the AI component tier. Every component, including dify, langflow, agent-exec, hermes-agent, and open-webui, now emits OpenTelemetry for every LLM call. So does the proxy and router tier that those components call through. Cribl Edge collects on the endpoint, Cribl Stream tags and fans out, and the spans land in Langfuse and Arize Phoenix. There are zero direct-to-backend paths. Splunk gets the same estate’s other telemetry, but not traces because they don’t fit its event model; see the fan-out default. These Cribl routing changes are MERGED in git; NOT DEPLOYED. Alert delivery through Stream is in ansible-proxmox-apps PR #2417 and nix-darwin PR #2740. The catalog is in ansible-proxmox-ai PR #1077 and nix-darwin PR #2737. Stream S2S span fan-out is in ansible-proxmox-apps PR #2426 (commit 728dc754). The S2S pack sends spans to Langfuse and Phoenix. It drops marked spans before the Splunk leg. Acceptance is pending the Stream converge. Observe 15 minutes with zero new span events in Splunk and a known trace in the trace backend.

​

Emitting traces: OpenLLMetry and OTEL GenAI

Apps are instrumented with OpenLLMetry (the Traceloop SDK). It wraps LLM providers, vector stores, and frameworks (LangChain, CrewAI). It emits spans following OpenTelemetry’s GenAI semantic conventions. Those conventions matured in 2026. Framework-native spans and SDK-emitted spans now line up on the same schema: prompt, completion, model, token counts, latency, cost. The spans leave the app over OTLP (gRPC 4317 / HTTP 4318) pointed at the collector, not at any one backend. Keeping the emit target on the pipeline, not the trace store, is what lets the same telemetry reach more than one place.

​

Cribl is the hub

A single collector tier owns intake and fan-out. Cribl Edge runs native OpenTelemetry sources, one per signal type on its own port, so it can route by type without parsing payloads. Edge relays to Cribl Stream, which tags each span with its producer and forks it in parallel:

• Langfuse gets the traces. It is the LLM-native view: trace waterfalls per request, token cost, prompt and completion inspection, plus prompt versioning. It is the trace-of-record.
• Phoenix gets the same traces on its own pipeline leg. It is where a trace becomes a test case: datasets, experiments, LLM-as-judge evals, and a playground that replays a captured call. See Arize Phoenix.
• Marked spans are dropped before the Splunk leg. The Stream S2S pack filters them before forwarding. Splunk still gets this estate’s other telemetry (metrics, logs) over its own legs.

Apps never talk to a trace store directly, and they never reach across into the monitoring tier. They emit to the collector, and the collector decides where it goes. One intake point, two trace sinks, no second collector to run.

​

Reaching the pipeline: subdomains, TLS, and who gets gated

Two rules govern how anything reaches a service in this stack, and both matter for telemetry. Every service answers on its own subdomain over TLS. A caller uses https://<app>.<subdomain>/ and nothing else. No caller targets a raw port, whether human, agent, or collector. The port a service listens on is an implementation detail of the host it runs on. A config that hardcodes the port breaks the moment the service moves. Naming the subdomain instead means a move costs a DNS record, not an edit in every producer. Single sign-on (SSO) gates humans, never ingestion. The forward-auth layer sits in front of the human-facing routes: dashboards, trace browsers, administrator consoles. Machine telemetry must not sit behind an interactive login. A collector cannot complete a browser redirect, and a pipeline that stalls on a login page drops data silently rather than failing loudly. You can tell the two apart by what an unauthenticated request gets back. Call an ingestion path with no credentials and the answer is the receiving service’s own 401. The service saw the request and rejected the credential. Call a human-facing path the same way and the answer is a redirect to the login flow. The request never reached the service at all. A 401 from the app is the signal that ingestion is correctly excluded from the human gate.

​

What the traces do not cover

Two limits are worth stating plainly, because a dashboard that looks complete invites the wrong conclusion. In-process tracing is not available for every serving backend. Some model servers ship no OpenTelemetry integration at all. Calls to those backends are still recorded. The proxy tier in front of them attributes the model, the token counts, and the latency, but the span stops at the proxy boundary. You see that the call happened and what it cost in time and tokens. You do not see inside the server: no per-stage timing, no queue depth, no batching detail. Coverage of calls is complete; coverage of internals is not. Per-call cost is only meaningful where a price exists. Cost is computed from a published per-token price for the model. A locally served model has no such price, so its cost is reported as zero. That zero is a statement about billing, not about resource use. The call still consumed power, memory, and time. Read cost figures as “what a vendor invoiced,” and never as “what this workload consumed.”

​

Client endpoint rule

Clients reach Langfuse through the ingress name, the same pattern as the other gated services. The bare service name resolves to a backend address on the service subnet. Only hosts on that subnet can reach that address directly.

• A workstation, or any host off the service subnet, gets a timeout on the bare name. The ingress name returns a healthy response from the same workstation.
• Gatus checks the backend from its own segment, so its green result does not prove a workstation can reach the service.
• Every config source in the estate composes the ingress name from the environment, so a client that sets the bare name is the exception to fix.
• Probes and trace checks from a workstation use the ingress name.

Decision: no DNS or routing change. Keep the ingress name in client config and in probes.

​

Why Langfuse

Laminar (Apache-2.0) is the runner-up. It’s lighter, tilted toward long-running agent debugging. Arize Phoenix ships under the Elastic License 2.0, which only forbids reselling it as a managed service. It runs here as the evals and experiments peer rather than as the trace-of-record.

​

Where to go next