Interactive · Evaluation methodology

Evidence explorer

Offline evaluation, randomized experiments, and progressive rollout answer three different questions. This page computes all three from inputs you control, so you can see where a decision actually comes from — and how easily a single favorable number can mislead.

Every estimate below is computed in your browser using the same procedures implemented in agent-evaluation-playbook. All values are illustrative.

Offline evaluation

LAYER 01

Can the feature do the task well, under a fixed rubric?

Evaluation tasks 200

Observed pass rate 86%

Release bar 80%

Passing tasks—

95% Wilson interval—

Lower bound vs. bar—

—

Online experiment

LAYER 02

Does the change cause a difference for real users, and does it harm anything we promised to protect?

Users per arm 4,000

Control success rate 42.0%

Treatment success rate 45.5%

Control guardrail-event rate 2.0%

Treatment guardrail-event rate 2.2%

Non-inferiority margin 1.0 pp

Absolute effect—

95% CI on the difference—

Minimum detectable effect (80% power)—

—

Guardrail difference—

95% upper bound on harm—

—

Progressive rollout

LAYER 03

At this exposure, is the evidence strong enough to widen the audience?

Sessions observed in this ring 600

Observed healthy-session rate 97.0%

Required health threshold 95.0%

Healthy sessions—

95% Wilson lower bound—

Sessions needed to clear the bar—

—

Combined recommendation

—

What this model does not capture

A calculator makes assumptions visible, which also makes its limits visible. The intervals above assume independent Bernoulli observations, a single pre-registered comparison, and a randomization unit that matches the analysis unit. Real evaluation work usually violates at least one of those.

Clustering. Sessions from the same user are correlated. Treating them as independent understates variance and produces intervals that are too narrow.
Multiplicity. Scanning slices, metrics, and time windows inflates the false-positive rate well beyond the nominal 5%.
Novelty and primacy. Early behavior after a visible change is often not the steady state, so a short window can measure reaction rather than value.
Non-random ring composition. Early rings are usually self-selected. A clean ring result is evidence about that ring, not an unbiased forecast of the next one.
Construct validity. A significant move in a weak proxy is still a weak claim. The hardest part is rarely the interval; it is arguing that the metric means what the decision needs it to mean.

The point of separating the three layers is that no single layer can be repaired by a larger sample in another. Offline evidence does not establish causality; an experiment does not establish operational safety at scale; a healthy ring does not establish that the feature is good.

Illustrative and general-purpose. Uses synthetic inputs only, and describes publicly discussable methodology rather than any specific product, system, or result. Views are my own.