Practice note · Evaluation
Separating capability, causal effect, and rollout risk
Offline evaluation, randomized experiments, and progressive rollout analysis estimate different quantities. Conflating them produces conclusions that the underlying measurements do not support.
An agentic feature rarely behaves like a conventional deterministic feature. Its output depends on the request, available context, tool state, retrieval quality, model behavior, and the path taken through the task. That flexibility is useful, but it makes a single aggregate metric a poor substitute for understanding.
I find it more useful to think about evaluation as a three-layer evidence stack. Each layer reduces a different uncertainty. Together, they support a decision. Separately, each one leaves a blind spot large enough to matter.
The three questions
Offline evaluation: can the feature do the task well?
Offline evaluation creates a controlled environment for fast iteration. It should include representative tasks, explicit rubrics, important slices, known failure modes, and enough observability to explain why a score moved.
Automated judges can extend coverage, but only after calibration against human labels. Judge agreement, bias by slice, sensitivity to verbosity, and drift after prompt or model changes are part of the evaluation system—not implementation details.
Online experimentation: does the feature help real users?
An offline win is evidence about a dataset and a rubric. An online experiment tests whether the change affects actual behavior under real traffic, context, latency, and user expectations.
A useful A/B test starts with an answerable question. The primary outcome, guardrail metrics, exposure logic, power assumptions, and analysis plan should follow from that question. Statistical significance does not rescue a weak proxy, and a neutral aggregate can hide meaningful movement in a critical population.
Progressive rollout: does the evidence remain stable as exposure grows?
Deployment rings are more than a release mechanism. They are a sequence of populations, operating conditions, and opportunities to discover failures that controlled tests did not represent.
Early rings are intentionally small, which makes their signals noisy. The right question is not whether every metric is significant. It is whether observed behavior is consistent with the expected mechanism, whether guardrails remain healthy, and whether new failure modes justify slowing down before broader exposure.
Why the layers disagree
Disagreement is not necessarily a measurement failure. It is often the most useful result. Offline quality can improve while online behavior stays flat because the evaluation set overrepresents a task, the metric rewards a property users do not value, or another system constraint dominates. An online metric can improve while qualitative failures remain unacceptable. A small ring can surface a severe issue that disappears in an aggregate.
The goal is not to make every metric agree. The goal is to understand what each metric knows, what it cannot know, and whether the combined evidence is strong enough for the next decision.
A decision-oriented workflow
- State the behavior change. Describe what should improve, for whom, and through what mechanism.
- Build the failure taxonomy. Define the errors that would make the feature unhelpful or unsafe.
- Choose evidence by uncertainty. Use offline tests for capability and diagnosis, online experiments for user impact, and rollout analysis for stability.
- Predefine guardrails. Decide what must not regress before looking at results.
- Read slices before averages. Agent behavior is rarely uniform across tasks, languages, or user populations.
- Record the decision, not only the score. A good evaluation system preserves why the team shipped, held, or iterated.
A decision criterion
An evidence-based deployment decision should support three claims:
- We understand the capability. Controlled evaluation shows where it works and fails.
- We observed user value. Online evidence supports the intended product outcome.
- We managed rollout risk. Progressive exposure did not reveal a reason to stop.
When one claim is weak, the next action should reduce that uncertainty—not manufacture confidence from a different layer.
Views are my own. This article discusses general evaluation methodology and does not describe confidential Microsoft product details, internal metrics, thresholds, or unreleased features.