A Demo Is Not an Evaluation: Testing AI Features Over Time

A demo shows that a system can succeed on a few selected examples. An evaluation checks whether it meets defined thresholds on representative, difficult and dangerous cases, then detects regressions when the model, prompt, data or workflow changes.

A fluent response and an enthusiastic meeting are not evidence of reliability.

Start from risk and outcome

Before selecting an evaluation tool, ask which outcome must remain true and which error is unacceptable.

A marketing rewrite feature and an assistant preparing a sensitive decision require different thresholds. Evaluation should follow impact, reversibility and recourse.

Build a case set

Include four families:

• Frequent cases: the core traffic, including real languages, formats and lengths.
• Difficult cases: incomplete information, ambiguity, contradiction, noise and domain exceptions.
• Dangerous cases: prompt injection, sensitive data, unauthorized action, stale sources and out-of-scope requests.
• Regression cases: every important incident becomes a reproducible test after sensitive data is removed or protected.

Start with 20 to 50 well-understood cases. A small set connected to decisions is better than a large opaque dataset.

Write criteria before seeing results

Possible criteria include factual accuracy, completeness, instruction following, citation, appropriate refusal, tone, cost, latency, ability to ask and security.

For each criterion, define the scoring method, passing threshold, critical failures that block release, owner of disagreements and review cadence.

A high average must not hide one catastrophic failure.

Three evaluator types

Deterministic rules

Check format, presence, schema, permissions or calculable values. They are fast and reproducible.

Model evaluators

Useful for scaling comparison of coherence, relevance or style. Calibrate them against human judgment. A number produced by a model is not automatically ground truth.

Human review

Essential for subtle criteria, domain context and high-impact cases. Measure disagreement between reviewers as well.

Combine the three instead of looking for one universal score.

Offline and online evaluation

Offline evaluation tests before exposure. Online evaluation observes reality: correction, recourse, abandonment, outcome and incident. A feature may pass offline cases and fail inside the workflow.

Connect every deployed version to its prompt, model, configuration, case set and result. Without versioning, quality changes are hard to explain.

Define an error budget

Set the acceptable amount of error for a period and segment, separating ordinary from critical failures. When the budget is consumed, slow feature change and improve reliability.

An error budget does not make severe harm acceptable. Some scenarios keep zero tolerance and require human approval.

After an incident

Protect the user, understand the journey and version, fix or roll back, add a regression case, review the control and degraded mode, then record the decision and outcome.

FAQ

Can every evaluation be automated?

No. Automation works for stable rules and helps sample. Human judgment remains necessary for context and impact.

How many cases are needed before a pilot?

Enough to cover the normal path, major exceptions and severe failures. Twenty well-chosen cases can start the process, then the set grows with use.

When should evaluations run again?

After every change to model, prompt, tool, source, policy or workflow, and periodically to detect drift.

Next step

Turn the last five errors or corrections into regression cases. Add five frequent and five dangerous cases, then write thresholds before testing. For a protocol review, email contact@yumea.fr with “EVALS” in the subject.

Sources

• NIST, AI Risk Management Framework.
• OWASP, Top 10 for LLM Applications 2025.
• Google SRE, Error Budget Policy.