Sentry, Traces, Logs and Evaluations: Which Question Does Each Answer?

When an AI feature fails, teams often open five tools and still cannot find the cause. The problem is not always missing data. Signals were not designed to answer distinct questions. Errors, traces, logs, evaluations and business events each have a role.

Application errors: did the software break?

A tool such as Sentry helps locate exceptions, regressions, affected releases, environments and user journeys. It answers: which code failure occurred, where and for how many users?

It does not fully answer whether a response was correct. A function can run without an exception and still produce a poor recommendation.

Traces: how did execution unfold?

A trace connects request stages: context retrieval, model call, external tool, retry, policy and response. It locates latency, failure and dependency problems.

For AI, add versions, delegation mode and source references. Avoid duplicating sensitive content by default.

Logs: which technical fact needs explanation?

A log records a discrete event useful for operations. It should be structured, limited and linked to the trace. Free-form messages accumulated without level, retention or owner become an expensive archive.

Do not confuse logging with business memory. A log operates the system; memory preserves a decision, rationale and outcome.

Evaluations: does the output meet the criteria?

An evaluation checks correctness, completeness, compliance, grounding, tone or a business rule. It can run offline on a case set or in production on a sample.

Automated judges also need calibration against human review. One overall score can hide degradation in an important segment.

Business events: did the user obtain the outcome?

Acceptance, correction, sending, resolution, abandonment and appeal connect technical quality to value. Without them, a team may optimize a score or latency without knowing whether the product helps.

The diagnostic map

Start with the observable complaint:

1. Broken screen or exception? Check errors and release.
2. Slow answer or missing tool? Follow the distributed trace.
3. Unexpected internal state? Inspect structured logs.
4. Fluent but wrong output? Run evaluations and inspect sources.
5. Correct output without value? Analyze the journey and business outcome.
6. Sensitive incident? Link trace, authority, decision, appeal and impact.

A shared correlation identifier prevents manual searching across layers.

Minimum instrumentation

For a first AI capability, instrument deployment version, latency, dependency status, model and policy versions, trace ID, critical evaluations, user correction and business outcome. Add signals only when they inform a decision.

Fact, inference and hypothesis

Direct fact: no exception does not imply semantic quality, and semantic quality does not imply business outcome.

Inference: separating signal layers reduces noise and speeds incident attribution.

Hypothesis to test: a shared diagnostic map reduces diagnosis time more than adding another tool.

FAQ

Should all signals be centralized?

No. They need reliable correlation, a known source of truth and appropriate access and retention.

Does an AI observability tool replace Sentry?

Not necessarily. It may add model traces and evaluations, but the real overlap should be assessed.

Can every prompt be retained for debugging?

Not by default. Use references, categories, hashes or protected samples under an explicit purpose.

Next step

Apply the diagnostic map to the last three incidents and record which signal was genuinely missing. For an observability architecture review, email contact@yumea.fr with the subject “AI DIAGNOSTICS.”